[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:08:33 dist-test-slave-mp03 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:08:33 dist-test-slave-mp03 krb5kdc[395](info): set up 2 sockets
Sep 27 22:08:33 dist-test-slave-mp03 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 22:08:34.199882 388 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:08:34 dist-test-slave-mp03 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:08:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:08:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:08:34.367841 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45365
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:08:34.687405 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:08:34.687837 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:08:34.688025 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:08:34.695335 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:08:34.695483 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:08:34.695591 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:08:34.695695 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:08:34.695783 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:08:34.705603 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:08:34.707027 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:08:34.708392 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:08:34.721089 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:34.721091 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:34.721171 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:08:34.721822 408 server_base.cc:1034] running on GCE node
I20260927 22:08:34.722582 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:08:34.724097 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:08:34.725298 408 hybrid_clock.cc:648] HybridClock initialized: now 1790546914725219 us; error 67 us; skew 500 ppm
I20260927 22:08:34.725653 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:08:34 dist-test-slave-mp03 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:08:34.729228 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:08:34.730728 408 webserver.cc:460] Webserver started at http://127.0.97.62:40479/ using document root <none> and password file <none>
I20260927 22:08:34.731383 408 fs_manager.cc:359] Metadata directory not provided
I20260927 22:08:34.731477 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:08:34.731724 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:08:34.734748 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "2f9e1cd25a17477e869f9375c1cbd5b0"
format_stamp: "Formatted at 2026-09-27 22:08:34 on dist-test-slave-mp03"
server_key: "b3f5dec8dc2c0b669a65264876f3fcc4"
server_key_iv: "7ad1280df3fbba5963427ae757eeadb6"
server_key_version: "encryptionkey@0"
I20260927 22:08:34.735463 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "2f9e1cd25a17477e869f9375c1cbd5b0"
format_stamp: "Formatted at 2026-09-27 22:08:34 on dist-test-slave-mp03"
server_key: "b3f5dec8dc2c0b669a65264876f3fcc4"
server_key_iv: "7ad1280df3fbba5963427ae757eeadb6"
server_key_version: "encryptionkey@0"
I20260927 22:08:34.740594 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:08:34.743939 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:34.745208 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:08:34.745378 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "2f9e1cd25a17477e869f9375c1cbd5b0"
format_stamp: "Formatted at 2026-09-27 22:08:34 on dist-test-slave-mp03"
server_key: "b3f5dec8dc2c0b669a65264876f3fcc4"
server_key_iv: "7ad1280df3fbba5963427ae757eeadb6"
server_key_version: "encryptionkey@0"
I20260927 22:08:34.745556 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:08:34.768450 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:08:34.772294 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:08:34.772603 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:08:34.798357 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45365
I20260927 22:08:34.798148 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45365 every 8 connection(s)
I20260927 22:08:34.801296 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:08:34.804100 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 408
I20260927 22:08:34.804394 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:08:34.806309 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:34.806785 388 external_mini_cluster.cc:1428] Setting key 99dff4e2f606214cb04f0c625cd9d6ee
I20260927 22:08:34.821411 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f9e1cd25a17477e869f9375c1cbd5b0: Bootstrap starting.
I20260927 22:08:34.825945 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f9e1cd25a17477e869f9375c1cbd5b0: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:34.827446 475 log.cc:824] T 00000000000000000000000000000000 P 2f9e1cd25a17477e869f9375c1cbd5b0: Log is configured to *not* fsync() on all Append() calls
I20260927 22:08:34.831279 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f9e1cd25a17477e869f9375c1cbd5b0: No bootstrap required, opened a new log
Sep 27 22:08:34 dist-test-slave-mp03 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:08:34.844225 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f9e1cd25a17477e869f9375c1cbd5b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9e1cd25a17477e869f9375c1cbd5b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45365 } }
I20260927 22:08:34.844653 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f9e1cd25a17477e869f9375c1cbd5b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:34.844782 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f9e1cd25a17477e869f9375c1cbd5b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f9e1cd25a17477e869f9375c1cbd5b0, State: Initialized, Role: FOLLOWER
I20260927 22:08:34.845440 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f9e1cd25a17477e869f9375c1cbd5b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9e1cd25a17477e869f9375c1cbd5b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45365 } }
I20260927 22:08:34.845701 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f9e1cd25a17477e869f9375c1cbd5b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:08:34.845865 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f9e1cd25a17477e869f9375c1cbd5b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:08:34.846050 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f9e1cd25a17477e869f9375c1cbd5b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:34.848372 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f9e1cd25a17477e869f9375c1cbd5b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9e1cd25a17477e869f9375c1cbd5b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45365 } }
I20260927 22:08:34.848892 475 leader_election.cc:302] T 00000000000000000000000000000000 P 2f9e1cd25a17477e869f9375c1cbd5b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f9e1cd25a17477e869f9375c1cbd5b0; no voters:
I20260927 22:08:34.849541 475 leader_election.cc:288] T 00000000000000000000000000000000 P 2f9e1cd25a17477e869f9375c1cbd5b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:08:34.849647 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f9e1cd25a17477e869f9375c1cbd5b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:08:34.852092 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f9e1cd25a17477e869f9375c1cbd5b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:08:34.852389 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:34.810949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54302 (local address 127.0.97.62:45365)
0927 22:08:34.811830 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:34.811850 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:34.813233 (+ 1383us) server_negotiation.cc:407] Connection header received
0927 22:08:34.825136 (+ 11903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:34.825166 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:34.825835 (+ 669us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:34.826380 (+ 545us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:34.830929 (+ 4549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:34.832481 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:34.835060 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:34.835752 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:34.835979 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:34.842939 (+ 6960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:34.842950 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:34.842971 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:34.843017 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:34.846574 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:34.847441 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:34.847450 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:34.847459 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:34.847641 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:34.848103 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:34.848108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:34.848113 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:34.848773 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:34.849085 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:34.850913 (+ 1828us) server_negotiation.cc:299] Negotiation successful
0927 22:08:34.851329 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":392,"threads_started":1}
I20260927 22:08:34.856890 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f9e1cd25a17477e869f9375c1cbd5b0 [term 1 LEADER]: Becoming Leader. State: Replica: 2f9e1cd25a17477e869f9375c1cbd5b0, State: Running, Role: LEADER
I20260927 22:08:34.857575 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f9e1cd25a17477e869f9375c1cbd5b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9e1cd25a17477e869f9375c1cbd5b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45365 } }
I20260927 22:08:34.860733 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f9e1cd25a17477e869f9375c1cbd5b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f9e1cd25a17477e869f9375c1cbd5b0. Latest consensus state: current_term: 1 leader_uuid: "2f9e1cd25a17477e869f9375c1cbd5b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9e1cd25a17477e869f9375c1cbd5b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45365 } } }
I20260927 22:08:34.861141 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f9e1cd25a17477e869f9375c1cbd5b0 [sys.catalog]: This master's current role is: LEADER
I20260927 22:08:34.860991 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f9e1cd25a17477e869f9375c1cbd5b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f9e1cd25a17477e869f9375c1cbd5b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9e1cd25a17477e869f9375c1cbd5b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45365 } } }
I20260927 22:08:34.861294 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f9e1cd25a17477e869f9375c1cbd5b0 [sys.catalog]: This master's current role is: LEADER
W20260927 22:08:34.868510 494 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2f9e1cd25a17477e869f9375c1cbd5b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:08:34.868638 494 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:08:34.868782 495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:08:34.870455 495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:08:34.879153 495 catalog_manager.cc:1347] Generated new cluster ID: 628622dd0ddd4adfb0bf4df52df5eacf
I20260927 22:08:34.879282 495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:08:34.902709 495 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:08:34.904172 495 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:08:34.919492 495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f9e1cd25a17477e869f9375c1cbd5b0: Generated new TSK 0
I20260927 22:08:34.920640 495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:08:34.972319 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45365
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:08:35.314491 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:08:35.314800 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:08:35.314896 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:08:35.322497 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:08:35.322674 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:08:35.322805 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:08:35.332576 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:08:35.333834 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:08:35.335093 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:08:35.348462 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:35.348362 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:35.348328 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:08:35.349608 501 server_base.cc:1034] running on GCE node
I20260927 22:08:35.350234 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:08:35.351260 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:08:35.352451 501 hybrid_clock.cc:648] HybridClock initialized: now 1790546915352388 us; error 55 us; skew 500 ppm
I20260927 22:08:35.352798 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:08:35 dist-test-slave-mp03 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:08:35.356208 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:08:35.357648 501 webserver.cc:460] Webserver started at http://127.0.97.1:35073/ using document root <none> and password file <none>
I20260927 22:08:35.358338 501 fs_manager.cc:359] Metadata directory not provided
I20260927 22:08:35.358469 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:08:35.358847 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:08:35.362051 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "169d573d50b9403b8b48bc07c545e610"
format_stamp: "Formatted at 2026-09-27 22:08:35 on dist-test-slave-mp03"
server_key: "4dad4597f568b6936f0c7a8aa6de64c4"
server_key_iv: "572ccaef308e19e3618235653f2f542e"
server_key_version: "encryptionkey@0"
I20260927 22:08:35.362885 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "169d573d50b9403b8b48bc07c545e610"
format_stamp: "Formatted at 2026-09-27 22:08:35 on dist-test-slave-mp03"
server_key: "4dad4597f568b6936f0c7a8aa6de64c4"
server_key_iv: "572ccaef308e19e3618235653f2f542e"
server_key_version: "encryptionkey@0"
I20260927 22:08:35.368155 501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 22:08:35.371596 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:35.372928 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:08:35.373102 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "169d573d50b9403b8b48bc07c545e610"
format_stamp: "Formatted at 2026-09-27 22:08:35 on dist-test-slave-mp03"
server_key: "4dad4597f568b6936f0c7a8aa6de64c4"
server_key_iv: "572ccaef308e19e3618235653f2f542e"
server_key_version: "encryptionkey@0"
I20260927 22:08:35.373301 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:08:35.394707 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:08:35.399281 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:08:35.399586 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:08:35.400555 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:08:35.401870 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:08:35.401975 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:35.402118 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:08:35.402189 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:35.438427 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37295
I20260927 22:08:35.438476 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37295 every 8 connection(s)
I20260927 22:08:35.439792 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:08:35.440469 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 501
I20260927 22:08:35.440718 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:08:35.441455 388 external_mini_cluster.cc:1428] Setting key 67876fbddf429cb9452650a08cf44eee
Sep 27 22:08:35 dist-test-slave-mp03 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 22:08:35.461104 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:35.442639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54493 (local address 127.0.97.62:45365)
0927 22:08:35.443109 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:35.443115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:35.444253 (+ 1138us) server_negotiation.cc:407] Connection header received
0927 22:08:35.446268 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:35.446273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:35.446356 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:35.446458 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:35.448002 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:35.448809 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:35.450665 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:35.451076 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:35.451209 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:35.455184 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:35.455189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:35.455192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:35.455250 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:35.458044 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:35.458831 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:35.458835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:35.458838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:35.458945 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:35.459367 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:35.459372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:35.459375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:35.459684 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:35.459834 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:35.460532 (+ 698us) server_negotiation.cc:299] Negotiation successful
0927 22:08:35.460738 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":223,"threads_started":1}
I20260927 22:08:35.462317 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:35.443334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.1:54493)
0927 22:08:35.444101 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:35.444143 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:35.446001 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:35.446743 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:35.446754 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:35.447273 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:08:35.447828 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:35.447846 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:35.448965 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:35.448970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:35.450495 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:35.450506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:35.451316 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:35.451323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:35.451563 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:35.452330 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:08:35.452371 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:08:35.454903 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:08:35.458188 (+ 3285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:35.458199 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:35.458217 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:35.458702 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:35.459051 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:35.459056 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:35.459060 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:35.459252 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:35.459819 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:08:35.459827 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:08:35.460157 (+ 330us) client_negotiation.cc:769] Sending connection context
0927 22:08:35.460429 (+ 272us) client_negotiation.cc:240] Negotiation successful
0927 22:08:35.460956 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":330,"threads_started":1}
I20260927 22:08:35.463913 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45365
I20260927 22:08:35.464223 629 heartbeater.cc:464] Registering TS with master...
I20260927 22:08:35.464920 629 heartbeater.cc:511] Master 127.0.97.62:45365 requested a full tablet report, sending...
I20260927 22:08:35.467000 440 ts_manager.cc:194] Registered new tserver with Master: 169d573d50b9403b8b48bc07c545e610 (127.0.97.1:37295)
I20260927 22:08:35.468441 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:08:35.482133 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45365
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:08:35.810451 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:08:35.810786 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:08:35.810969 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:08:35.821709 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:08:35.821816 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:08:35.821998 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:08:35.832671 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:08:35.833981 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:08:35.835388 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:08:35.848214 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:35.848737 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:35.849205 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:08:35.850112 635 server_base.cc:1034] running on GCE node
I20260927 22:08:35.850826 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:08:35.851744 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:08:35.852975 635 hybrid_clock.cc:648] HybridClock initialized: now 1790546915852935 us; error 53 us; skew 500 ppm
I20260927 22:08:35.853353 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:08:35 dist-test-slave-mp03 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:08:35.856482 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:08:35.857910 635 webserver.cc:460] Webserver started at http://127.0.97.2:39389/ using document root <none> and password file <none>
I20260927 22:08:35.858639 635 fs_manager.cc:359] Metadata directory not provided
I20260927 22:08:35.858788 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:08:35.859104 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:08:35.861956 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "ac6ee55ba6b84e88b71dbcc759e947cc"
format_stamp: "Formatted at 2026-09-27 22:08:35 on dist-test-slave-mp03"
server_key: "b80e1f96fec030163993c42ac4008f68"
server_key_iv: "4ebb60ee3d7548c97d98bc82c0dfce46"
server_key_version: "encryptionkey@0"
I20260927 22:08:35.862977 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ac6ee55ba6b84e88b71dbcc759e947cc"
format_stamp: "Formatted at 2026-09-27 22:08:35 on dist-test-slave-mp03"
server_key: "b80e1f96fec030163993c42ac4008f68"
server_key_iv: "4ebb60ee3d7548c97d98bc82c0dfce46"
server_key_version: "encryptionkey@0"
I20260927 22:08:35.868198 635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:08:35.872167 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:35.873504 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:08:35.873682 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "ac6ee55ba6b84e88b71dbcc759e947cc"
format_stamp: "Formatted at 2026-09-27 22:08:35 on dist-test-slave-mp03"
server_key: "b80e1f96fec030163993c42ac4008f68"
server_key_iv: "4ebb60ee3d7548c97d98bc82c0dfce46"
server_key_version: "encryptionkey@0"
I20260927 22:08:35.873893 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:08:35.905049 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:08:35.909487 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:08:35.909829 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:08:35.910944 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:08:35.912395 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:08:35.912513 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:35.912680 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:08:35.912772 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:35.950656 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36239
I20260927 22:08:35.950702 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36239 every 8 connection(s)
I20260927 22:08:35.952067 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:08:35.960075 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 635
I20260927 22:08:35.960278 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:08:35.960789 388 external_mini_cluster.cc:1428] Setting key 922435bcd4ea1a3c13b9ee00ee2aa542
Sep 27 22:08:35 dist-test-slave-mp03 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:08:35.976240 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:35.954671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54857 (local address 127.0.97.62:45365)
0927 22:08:35.954858 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:35.954863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:35.956074 (+ 1211us) server_negotiation.cc:407] Connection header received
0927 22:08:35.958550 (+ 2476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:35.958556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:35.958682 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:35.958790 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:35.960617 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:35.961405 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:35.963476 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:35.964604 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:35.964773 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:35.968818 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:35.968828 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:35.968836 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:35.968881 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:35.972879 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:35.973601 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:35.973609 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:35.973616 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:35.973761 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:35.974182 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:35.974190 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:35.974196 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:35.974642 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:35.974797 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:35.975493 (+ 696us) server_negotiation.cc:299] Negotiation successful
0927 22:08:35.975778 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:08:35.976915 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:35.955170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.2:54857)
0927 22:08:35.955906 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:35.955942 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:35.958272 (+ 2330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:35.959038 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:35.959046 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:35.959561 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:08:35.960434 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:35.960460 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:35.961563 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:35.961570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:35.963328 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:35.963340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:35.964819 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:35.964824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:35.965035 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:35.965768 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:08:35.965790 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:08:35.968578 (+ 2788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:08:35.973028 (+ 4450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:35.973036 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:35.973050 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:35.973465 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:35.973876 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:35.973879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:35.973881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:35.974058 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:35.974775 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:08:35.974782 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:08:35.975101 (+ 319us) client_negotiation.cc:769] Sending connection context
0927 22:08:35.975330 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 22:08:35.975715 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":272,"threads_started":1}
I20260927 22:08:35.978629 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45365
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:08:35.978979 763 heartbeater.cc:464] Registering TS with master...
I20260927 22:08:35.979676 763 heartbeater.cc:511] Master 127.0.97.62:45365 requested a full tablet report, sending...
I20260927 22:08:35.981137 440 ts_manager.cc:194] Registered new tserver with Master: ac6ee55ba6b84e88b71dbcc759e947cc (127.0.97.2:36239)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 22:08:35.982079 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:08:36.002372 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45365
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:08:36.332392 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:08:36.332677 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:08:36.332836 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:08:36.340131 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:08:36.340265 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:08:36.340457 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:08:36.351280 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:08:36.352674 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:08:36.354074 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:08:36.368009 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:36.367969 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:36.367972 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:08:36.368738 768 server_base.cc:1034] running on GCE node
I20260927 22:08:36.369264 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:08:36.370106 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:08:36.371294 768 hybrid_clock.cc:648] HybridClock initialized: now 1790546916371225 us; error 59 us; skew 500 ppm
I20260927 22:08:36.371624 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:08:36 dist-test-slave-mp03 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:08:36.374574 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:08:36.375978 768 webserver.cc:460] Webserver started at http://127.0.97.3:39713/ using document root <none> and password file <none>
I20260927 22:08:36.376747 768 fs_manager.cc:359] Metadata directory not provided
I20260927 22:08:36.376900 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:08:36.377211 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:08:36.380311 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "599f7054a9494717b5fa552f424f3867"
format_stamp: "Formatted at 2026-09-27 22:08:36 on dist-test-slave-mp03"
server_key: "b66fbade66a870f9a00d741e4449a860"
server_key_iv: "9a7a8ee3eaa75c518ce2b5825003c052"
server_key_version: "encryptionkey@0"
I20260927 22:08:36.381072 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "599f7054a9494717b5fa552f424f3867"
format_stamp: "Formatted at 2026-09-27 22:08:36 on dist-test-slave-mp03"
server_key: "b66fbade66a870f9a00d741e4449a860"
server_key_iv: "9a7a8ee3eaa75c518ce2b5825003c052"
server_key_version: "encryptionkey@0"
I20260927 22:08:36.386430 768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 22:08:36.389720 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:36.391004 768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:08:36.391162 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "599f7054a9494717b5fa552f424f3867"
format_stamp: "Formatted at 2026-09-27 22:08:36 on dist-test-slave-mp03"
server_key: "b66fbade66a870f9a00d741e4449a860"
server_key_iv: "9a7a8ee3eaa75c518ce2b5825003c052"
server_key_version: "encryptionkey@0"
I20260927 22:08:36.391382 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:08:36.411140 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:08:36.415261 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:08:36.415576 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:08:36.416567 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:08:36.418421 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:08:36.418532 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:36.418720 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:08:36.418802 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:36.457000 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37705
I20260927 22:08:36.457032 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37705 every 8 connection(s)
I20260927 22:08:36.458451 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:08:36.459404 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 768
I20260927 22:08:36.459583 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:08:36.460072 388 external_mini_cluster.cc:1428] Setting key 9c4590f44c825ad38a275e346e63824a
I20260927 22:08:36.471383 629 heartbeater.cc:503] Master 127.0.97.62:45365 was elected leader, sending a full tablet report...
Sep 27 22:08:36 dist-test-slave-mp03 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:08:36.480273 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:36.461452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47939 (local address 127.0.97.62:45365)
0927 22:08:36.461640 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:36.461647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:36.463456 (+ 1809us) server_negotiation.cc:407] Connection header received
0927 22:08:36.465659 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:36.465663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:36.465727 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:36.465808 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:36.467283 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:36.468030 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:36.469463 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:36.469832 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:36.469921 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:36.473817 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:36.473822 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:36.473825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:36.473861 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:36.477097 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:36.477818 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:36.477822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:36.477824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:36.477985 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:36.478442 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:36.478446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:36.478450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:36.478801 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:36.478967 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:36.479658 (+ 691us) server_negotiation.cc:299] Negotiation successful
0927 22:08:36.479862 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:08:36.481261 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:36.462026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45365 (local address 127.0.97.3:47939)
0927 22:08:36.462871 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:36.462914 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:36.465404 (+ 2490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:36.465989 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:36.465997 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:36.466511 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:08:36.467144 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:36.467161 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:36.468146 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:36.468149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:36.469341 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:36.469351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:36.470013 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:36.470017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:36.470206 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:36.470908 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:08:36.470935 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:08:36.473646 (+ 2711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:08:36.477214 (+ 3568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:36.477255 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:36.477273 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:36.477703 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:36.478107 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:36.478112 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:36.478116 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:36.478320 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:36.478946 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:08:36.478954 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:08:36.479289 (+ 335us) client_negotiation.cc:769] Sending connection context
0927 22:08:36.479556 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 22:08:36.480097 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":362,"threads_started":1}
I20260927 22:08:36.482482 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45365
I20260927 22:08:36.482797 896 heartbeater.cc:464] Registering TS with master...
I20260927 22:08:36.483494 896 heartbeater.cc:511] Master 127.0.97.62:45365 requested a full tablet report, sending...
I20260927 22:08:36.484804 440 ts_manager.cc:194] Registered new tserver with Master: 599f7054a9494717b5fa552f424f3867 (127.0.97.3:37705)
I20260927 22:08:36.485554 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47939
I20260927 22:08:36.488498 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:08:36 dist-test-slave-mp03 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546916, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:08:36 dist-test-slave-mp03 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546916, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:08:36.532367 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:36.517583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49998 (local address 127.0.97.1:37295)
0927 22:08:36.518209 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:36.518216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:36.518243 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:08:36.518301 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:36.518307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:36.518526 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:36.518756 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:36.520105 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:36.521100 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:36.522331 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:36.522950 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:36.523082 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:36.527079 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:36.527089 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:36.527104 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:36.527134 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:36.529733 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:36.530361 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:36.530368 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:36.530370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:36.530492 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:36.530884 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:36.530890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:36.530895 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:36.531213 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:36.531514 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:36.531777 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 22:08:36.532031 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":336,"threads_started":1}
W20260927 22:08:36.533555 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49998
Sep 27 22:08:36 dist-test-slave-mp03 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:08:36 dist-test-slave-mp03 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:08:36.576457 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:36.563104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50010 (local address 127.0.97.1:37295)
0927 22:08:36.563328 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:36.563334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:36.563678 (+ 344us) server_negotiation.cc:407] Connection header received
0927 22:08:36.563896 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:36.563900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:36.563998 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:36.564110 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:36.565398 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:36.566096 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:36.567345 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:36.567742 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:36.567895 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:36.571565 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:36.571576 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:36.571580 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:36.571611 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:36.574004 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:36.574624 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:36.574629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:36.574633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:36.574756 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:36.575111 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:36.575118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:36.575121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:36.575370 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:36.575603 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:36.575895 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 22:08:36.576103 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:08:36.577951 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 501
I20260927 22:08:36.590900 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:36.579011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50014 (local address 127.0.97.1:37295)
0927 22:08:36.579190 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:36.579197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:36.579455 (+ 258us) server_negotiation.cc:407] Connection header received
0927 22:08:36.579634 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:36.579640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:36.579704 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:36.579796 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:36.580914 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:36.581585 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:36.582839 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:36.583244 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:36.583373 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:36.585543 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:36.585549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:36.585554 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:36.585592 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:36.588405 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:36.589034 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:36.589039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:36.589043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:36.589171 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:36.589525 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:36.589532 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:36.589535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:36.589781 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:36.590023 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:36.590309 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 22:08:36.590511 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 22:08:36.591588 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:08:36.673861 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 501
I20260927 22:08:36.690071 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 635
Sep 27 22:08:36 dist-test-slave-mp03 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:08:36.705217 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:36.690966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51676 (local address 127.0.97.2:36239)
0927 22:08:36.691496 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:36.691509 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:36.691534 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:08:36.691600 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:36.691606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:36.691804 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:36.691943 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:36.693012 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:36.693984 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:36.695344 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:36.695926 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:36.696021 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:36.699858 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:36.699867 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:36.699881 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:36.699921 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:36.702893 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:36.703427 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:36.703433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:36.703435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:36.703543 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:36.703887 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:36.703890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:36.703892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:36.704283 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:36.704501 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:36.704723 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 22:08:36.704902 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":282,"threads_started":1}
I20260927 22:08:36.706463 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:08:36.788504 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 635
I20260927 22:08:36.815233 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 768
Sep 27 22:08:36 dist-test-slave-mp03 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:08:36.830902 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:36.816217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51588 (local address 127.0.97.3:37705)
0927 22:08:36.816817 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:36.816824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:36.816851 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:08:36.816925 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:36.816931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:36.817130 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:36.817299 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:36.818524 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:36.819604 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:36.820825 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:36.821375 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:36.821504 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:36.825625 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:36.825636 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:36.825651 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:36.825688 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:36.828493 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:36.829121 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:36.829127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:36.829129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:36.829239 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:36.829549 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:36.829552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:36.829554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:36.829866 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:36.830076 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:36.830368 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 22:08:36.830565 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":321,"threads_started":1}
I20260927 22:08:36.832146 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:08:36.914531 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 768
I20260927 22:08:36.930761 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 408
I20260927 22:08:36.931969 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:08:37.033602 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 408
2026-09-27T22:08:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4014 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:08:37 dist-test-slave-mp03 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:08:37 dist-test-slave-mp03 krb5kdc[917](info): set up 2 sockets
Sep 27 22:08:37 dist-test-slave-mp03 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 22:08:38.118515 388 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:08:38 dist-test-slave-mp03 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:08:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:08:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:08:38.290144 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36829
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:08:38.603583 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:08:38.603892 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:08:38.604050 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:08:38.611356 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:08:38.611474 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:08:38.611521 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:08:38.611613 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:08:38.611694 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:08:38.621609 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:08:38.623005 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:08:38.624347 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:08:38.635999 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:38.636157 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:38.636179 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:08:38.637253 930 server_base.cc:1034] running on GCE node
I20260927 22:08:38.637991 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:08:38.639531 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:08:38.640759 930 hybrid_clock.cc:648] HybridClock initialized: now 1790546918640710 us; error 43 us; skew 500 ppm
I20260927 22:08:38.641089 930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:08:38 dist-test-slave-mp03 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:08:38.651958 930 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:08:38.653612 930 webserver.cc:460] Webserver started at http://127.0.97.62:35655/ using document root <none> and password file <none>
I20260927 22:08:38.654335 930 fs_manager.cc:359] Metadata directory not provided
I20260927 22:08:38.654481 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:08:38.654863 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:08:38.657884 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "d462429634334fd1887b39eb1f6d4938"
format_stamp: "Formatted at 2026-09-27 22:08:38 on dist-test-slave-mp03"
server_key: "37396c4b1592a765703b44bc10441370"
server_key_iv: "f37fb2bdd188f093863c4e921d7511c2"
server_key_version: "encryptionkey@0"
I20260927 22:08:38.658703 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "d462429634334fd1887b39eb1f6d4938"
format_stamp: "Formatted at 2026-09-27 22:08:38 on dist-test-slave-mp03"
server_key: "37396c4b1592a765703b44bc10441370"
server_key_iv: "f37fb2bdd188f093863c4e921d7511c2"
server_key_version: "encryptionkey@0"
I20260927 22:08:38.663833 930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:08:38.667255 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:38.668627 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:08:38.668809 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "d462429634334fd1887b39eb1f6d4938"
format_stamp: "Formatted at 2026-09-27 22:08:38 on dist-test-slave-mp03"
server_key: "37396c4b1592a765703b44bc10441370"
server_key_iv: "f37fb2bdd188f093863c4e921d7511c2"
server_key_version: "encryptionkey@0"
I20260927 22:08:38.669049 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:08:38.696770 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:08:38.700839 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:08:38.701136 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:08:38.719599 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36829
I20260927 22:08:38.719646 996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36829 every 8 connection(s)
I20260927 22:08:38.720999 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:08:38.724647 997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:38.726857 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 930
I20260927 22:08:38.727074 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:08:38.727777 388 external_mini_cluster.cc:1428] Setting key 1d1346613fb88d4f5a116e963a6e395a
I20260927 22:08:38.736428 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d462429634334fd1887b39eb1f6d4938: Bootstrap starting.
I20260927 22:08:38.740180 997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d462429634334fd1887b39eb1f6d4938: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:38.741466 997 log.cc:824] T 00000000000000000000000000000000 P d462429634334fd1887b39eb1f6d4938: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:08:38 dist-test-slave-mp03 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:08:38.744697 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d462429634334fd1887b39eb1f6d4938: No bootstrap required, opened a new log
I20260927 22:08:38.751848 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:38.729839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39600 (local address 127.0.97.62:36829)
0927 22:08:38.730703 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:38.730715 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:38.730757 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:08:38.732385 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:38.732406 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:38.732807 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:38.733129 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:38.734853 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:38.736134 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:38.737837 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:38.738494 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:38.738695 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:38.743261 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:38.743272 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:38.743289 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:38.743331 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:38.747253 (+ 3922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:38.747926 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:38.747935 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:38.747944 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:38.748157 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:38.748480 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:38.748485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:38.748489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:38.749217 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:38.749508 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:38.749906 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 22:08:38.750385 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":365,"threads_started":1}
I20260927 22:08:38.755110 997 raft_consensus.cc:348] T 00000000000000000000000000000000 P d462429634334fd1887b39eb1f6d4938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d462429634334fd1887b39eb1f6d4938" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36829 } }
I20260927 22:08:38.755426 997 raft_consensus.cc:374] T 00000000000000000000000000000000 P d462429634334fd1887b39eb1f6d4938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:38.755499 997 raft_consensus.cc:729] T 00000000000000000000000000000000 P d462429634334fd1887b39eb1f6d4938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d462429634334fd1887b39eb1f6d4938, State: Initialized, Role: FOLLOWER
I20260927 22:08:38.756029 997 consensus_queue.cc:260] T 00000000000000000000000000000000 P d462429634334fd1887b39eb1f6d4938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d462429634334fd1887b39eb1f6d4938" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36829 } }
I20260927 22:08:38.756232 997 raft_consensus.cc:388] T 00000000000000000000000000000000 P d462429634334fd1887b39eb1f6d4938 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:08:38.756323 997 raft_consensus.cc:482] T 00000000000000000000000000000000 P d462429634334fd1887b39eb1f6d4938 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:08:38.756444 997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d462429634334fd1887b39eb1f6d4938 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:38.758828 997 raft_consensus.cc:504] T 00000000000000000000000000000000 P d462429634334fd1887b39eb1f6d4938 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d462429634334fd1887b39eb1f6d4938" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36829 } }
I20260927 22:08:38.759258 997 leader_election.cc:302] T 00000000000000000000000000000000 P d462429634334fd1887b39eb1f6d4938 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d462429634334fd1887b39eb1f6d4938; no voters:
I20260927 22:08:38.759807 997 leader_election.cc:288] T 00000000000000000000000000000000 P d462429634334fd1887b39eb1f6d4938 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:08:38.759979 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d462429634334fd1887b39eb1f6d4938 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:08:38.760643 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P d462429634334fd1887b39eb1f6d4938 [term 1 LEADER]: Becoming Leader. State: Replica: d462429634334fd1887b39eb1f6d4938, State: Running, Role: LEADER
I20260927 22:08:38.761108 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P d462429634334fd1887b39eb1f6d4938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d462429634334fd1887b39eb1f6d4938" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36829 } }
I20260927 22:08:38.761821 997 sys_catalog.cc:564] T 00000000000000000000000000000000 P d462429634334fd1887b39eb1f6d4938 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:08:38.764175 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P d462429634334fd1887b39eb1f6d4938 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d462429634334fd1887b39eb1f6d4938. Latest consensus state: current_term: 1 leader_uuid: "d462429634334fd1887b39eb1f6d4938" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d462429634334fd1887b39eb1f6d4938" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36829 } } }
I20260927 22:08:38.764189 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P d462429634334fd1887b39eb1f6d4938 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d462429634334fd1887b39eb1f6d4938" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d462429634334fd1887b39eb1f6d4938" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36829 } } }
I20260927 22:08:38.764583 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P d462429634334fd1887b39eb1f6d4938 [sys.catalog]: This master's current role is: LEADER
I20260927 22:08:38.764647 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P d462429634334fd1887b39eb1f6d4938 [sys.catalog]: This master's current role is: LEADER
I20260927 22:08:38.767076 1009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:08:38.772140 1009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:08:38.780400 1009 catalog_manager.cc:1347] Generated new cluster ID: 2eb81d19af794817a87c1b3c495e545b
I20260927 22:08:38.780532 1009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:08:38.819330 1009 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:08:38.821264 1009 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:08:38.833119 1009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d462429634334fd1887b39eb1f6d4938: Generated new TSK 0
I20260927 22:08:38.834219 1009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:08:38.882211 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36829
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:08:39.209896 1023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:08:39.210162 1023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:08:39.210332 1023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:08:39.217657 1023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:08:39.217782 1023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:08:39.217967 1023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:08:39.227856 1023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36829
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:08:39.229164 1023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:08:39.230526 1023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:08:39.243729 1028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:39.243778 1029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:39.243826 1031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:08:39.245150 1023 server_base.cc:1034] running on GCE node
I20260927 22:08:39.245679 1023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:08:39.246498 1023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:08:39.247710 1023 hybrid_clock.cc:648] HybridClock initialized: now 1790546919247644 us; error 40 us; skew 500 ppm
I20260927 22:08:39.248031 1023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:08:39 dist-test-slave-mp03 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:08:39.251260 1023 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:08:39.252615 1023 webserver.cc:460] Webserver started at http://127.0.97.1:37817/ using document root <none> and password file <none>
I20260927 22:08:39.253237 1023 fs_manager.cc:359] Metadata directory not provided
I20260927 22:08:39.253334 1023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:08:39.253592 1023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:08:39.256425 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "1c5480b12bc8496fb03fc7d87ba0db69"
format_stamp: "Formatted at 2026-09-27 22:08:39 on dist-test-slave-mp03"
server_key: "1a4d729c532697a3756b9e4fca4d7704"
server_key_iv: "6cca3438169b2e97071617376a89d7a1"
server_key_version: "encryptionkey@0"
I20260927 22:08:39.257225 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1c5480b12bc8496fb03fc7d87ba0db69"
format_stamp: "Formatted at 2026-09-27 22:08:39 on dist-test-slave-mp03"
server_key: "1a4d729c532697a3756b9e4fca4d7704"
server_key_iv: "6cca3438169b2e97071617376a89d7a1"
server_key_version: "encryptionkey@0"
I20260927 22:08:39.262243 1023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:08:39.265786 1038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:39.267174 1023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:08:39.267328 1023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "1c5480b12bc8496fb03fc7d87ba0db69"
format_stamp: "Formatted at 2026-09-27 22:08:39 on dist-test-slave-mp03"
server_key: "1a4d729c532697a3756b9e4fca4d7704"
server_key_iv: "6cca3438169b2e97071617376a89d7a1"
server_key_version: "encryptionkey@0"
I20260927 22:08:39.267521 1023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:08:39.290905 1023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:08:39.295266 1023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:08:39.295562 1023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:08:39.296500 1023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:08:39.297777 1023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:08:39.297845 1023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:39.298002 1023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:08:39.298069 1023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:39.330467 1023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43287
I20260927 22:08:39.330520 1150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43287 every 8 connection(s)
I20260927 22:08:39.331789 1023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:08:39.339699 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 1023
I20260927 22:08:39.339947 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:08:39.340677 388 external_mini_cluster.cc:1428] Setting key 306758b6790cbd895f41b465e0675d2e
Sep 27 22:08:39 dist-test-slave-mp03 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:08:39.352193 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:39.334465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34263 (local address 127.0.97.62:36829)
0927 22:08:39.334914 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:39.334919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:39.335858 (+ 939us) server_negotiation.cc:407] Connection header received
0927 22:08:39.337630 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:39.337635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:39.337705 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:39.337803 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:39.339320 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:39.340449 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:39.342270 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:39.342816 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:39.342953 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:39.346190 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:39.346199 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:39.346207 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:39.346260 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:39.348963 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:39.349879 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:39.349883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:39.349885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:39.349991 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:39.350449 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:39.350453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:39.350455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:39.350785 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:39.350894 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:39.351788 (+ 894us) server_negotiation.cc:299] Negotiation successful
0927 22:08:39.351954 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":269,"threads_started":1}
I20260927 22:08:39.353886 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:39.334969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36829 (local address 127.0.97.1:34263)
0927 22:08:39.335666 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:39.335720 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:39.337373 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:39.337994 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:39.338002 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:39.338499 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:08:39.339117 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:39.339133 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:39.340628 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:39.340631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:39.342073 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:39.342085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:39.343012 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:39.343016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:39.343220 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:39.343902 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:08:39.343924 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:08:39.345981 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:08:39.349101 (+ 3120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:39.349113 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:39.349134 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:39.349768 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:39.350092 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:39.350099 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:39.350106 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:39.350348 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:39.350915 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:08:39.350926 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:08:39.351387 (+ 461us) client_negotiation.cc:769] Sending connection context
0927 22:08:39.351709 (+ 322us) client_negotiation.cc:240] Negotiation successful
0927 22:08:39.352344 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":271,"threads_started":1}
I20260927 22:08:39.355685 1151 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36829
I20260927 22:08:39.356000 1151 heartbeater.cc:464] Registering TS with master...
I20260927 22:08:39.356715 1151 heartbeater.cc:511] Master 127.0.97.62:36829 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:08:39.358938 962 ts_manager.cc:194] Registered new tserver with Master: 1c5480b12bc8496fb03fc7d87ba0db69 (127.0.97.1:43287)
I20260927 22:08:39.360354 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:08:39.381923 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36829
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:08:39.699492 1157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:08:39.699837 1157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:08:39.699982 1157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:08:39.707481 1157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:08:39.707594 1157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:08:39.707710 1157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:08:39.717483 1157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36829
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:08:39.718787 1157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:08:39.720563 1157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:08:39.735405 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:39.735455 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:39.735651 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:08:39.737080 1157 server_base.cc:1034] running on GCE node
I20260927 22:08:39.737677 1157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:08:39.738638 1157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:08:39.739863 1157 hybrid_clock.cc:648] HybridClock initialized: now 1790546919739803 us; error 57 us; skew 500 ppm
I20260927 22:08:39.740171 1157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:08:39 dist-test-slave-mp03 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:08:39.743191 1157 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:08:39.744582 1157 webserver.cc:460] Webserver started at http://127.0.97.2:45227/ using document root <none> and password file <none>
I20260927 22:08:39.745260 1157 fs_manager.cc:359] Metadata directory not provided
I20260927 22:08:39.745388 1157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:08:39.745697 1157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:08:39.748574 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "7100e0b028c449bd8e2d48850ec6b012"
format_stamp: "Formatted at 2026-09-27 22:08:39 on dist-test-slave-mp03"
server_key: "2ba22b09a4ac106e69a859090b0db045"
server_key_iv: "2be95db79525a83649e103692ea92499"
server_key_version: "encryptionkey@0"
I20260927 22:08:39.749315 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7100e0b028c449bd8e2d48850ec6b012"
format_stamp: "Formatted at 2026-09-27 22:08:39 on dist-test-slave-mp03"
server_key: "2ba22b09a4ac106e69a859090b0db045"
server_key_iv: "2be95db79525a83649e103692ea92499"
server_key_version: "encryptionkey@0"
I20260927 22:08:39.754230 1157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 22:08:39.757519 1172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:39.758831 1157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:08:39.758987 1157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "7100e0b028c449bd8e2d48850ec6b012"
format_stamp: "Formatted at 2026-09-27 22:08:39 on dist-test-slave-mp03"
server_key: "2ba22b09a4ac106e69a859090b0db045"
server_key_iv: "2be95db79525a83649e103692ea92499"
server_key_version: "encryptionkey@0"
I20260927 22:08:39.759174 1157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:08:39.772996 1157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:08:39.776827 1157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:08:39.777101 1157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:08:39.778030 1157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:08:39.779386 1157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:08:39.779484 1157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:39.779623 1157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:08:39.779699 1157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:39.811908 1157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38949
I20260927 22:08:39.811954 1284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38949 every 8 connection(s)
I20260927 22:08:39.813226 1157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:08:39.818377 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 1157
I20260927 22:08:39.818583 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:08:39.819039 388 external_mini_cluster.cc:1428] Setting key 018801238e863a444382732321279a6f
Sep 27 22:08:39 dist-test-slave-mp03 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:08:39.832844 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:39.815712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49909 (local address 127.0.97.62:36829)
0927 22:08:39.815901 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:39.815906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:39.817061 (+ 1155us) server_negotiation.cc:407] Connection header received
0927 22:08:39.819042 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:39.819047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:39.819165 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:39.819286 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:39.820999 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:39.821712 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:39.823321 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:39.823720 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:39.823859 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:39.827272 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:39.827277 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:39.827280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:39.827311 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:39.829805 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:39.830538 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:39.830548 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:39.830550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:39.830727 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:39.831162 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:39.831167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:39.831170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:39.831432 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:39.831580 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:39.832250 (+ 670us) server_negotiation.cc:299] Negotiation successful
0927 22:08:39.832476 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:08:39.834088 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:39.816223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36829 (local address 127.0.97.2:49909)
0927 22:08:39.816907 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:39.816943 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:39.818779 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:39.819527 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:39.819535 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:39.820180 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:08:39.820837 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:39.820856 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:39.821867 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:39.821877 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:39.823176 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:39.823187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:39.823944 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:39.823951 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:39.824170 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:39.824869 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:08:39.824891 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:08:39.827056 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:08:39.829956 (+ 2900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:39.829965 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:39.829982 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:39.830412 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:39.830842 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:39.830848 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:39.830854 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:39.831044 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:39.831554 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:08:39.831561 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:08:39.831903 (+ 342us) client_negotiation.cc:769] Sending connection context
0927 22:08:39.832160 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 22:08:39.832711 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":265,"threads_started":1}
I20260927 22:08:39.835558 1285 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36829
I20260927 22:08:39.835938 1285 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 22:08:39.836725 1285 heartbeater.cc:511] Master 127.0.97.62:36829 requested a full tablet report, sending...
I20260927 22:08:39.838112 962 ts_manager.cc:194] Registered new tserver with Master: 7100e0b028c449bd8e2d48850ec6b012 (127.0.97.2:38949)
I20260927 22:08:39.838994 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:08:39.858418 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36829
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:08:40.186836 1290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:08:40.187132 1290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:08:40.187229 1290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:08:40.194480 1290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:08:40.194615 1290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:08:40.194788 1290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:08:40.204815 1290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36829
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:08:40.206215 1290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:08:40.207511 1290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:08:40.220363 1295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:40.220366 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:40.220410 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:08:40.221621 1290 server_base.cc:1034] running on GCE node
I20260927 22:08:40.222178 1290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:08:40.223150 1290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:08:40.224337 1290 hybrid_clock.cc:648] HybridClock initialized: now 1790546920224279 us; error 55 us; skew 500 ppm
I20260927 22:08:40.224646 1290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:08:40 dist-test-slave-mp03 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:08:40.227814 1290 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:08:40.229182 1290 webserver.cc:460] Webserver started at http://127.0.97.3:34055/ using document root <none> and password file <none>
I20260927 22:08:40.229836 1290 fs_manager.cc:359] Metadata directory not provided
I20260927 22:08:40.229935 1290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:08:40.230180 1290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:08:40.233027 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "aaea47adf0df4902bca5240d55f32349"
format_stamp: "Formatted at 2026-09-27 22:08:40 on dist-test-slave-mp03"
server_key: "653e5ccf314cbe19ce75e732ca3eb8d4"
server_key_iv: "09512bf2ff30fa6bb900defd80214d22"
server_key_version: "encryptionkey@0"
I20260927 22:08:40.233736 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "aaea47adf0df4902bca5240d55f32349"
format_stamp: "Formatted at 2026-09-27 22:08:40 on dist-test-slave-mp03"
server_key: "653e5ccf314cbe19ce75e732ca3eb8d4"
server_key_iv: "09512bf2ff30fa6bb900defd80214d22"
server_key_version: "encryptionkey@0"
I20260927 22:08:40.238793 1290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 22:08:40.242329 1305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:40.243685 1290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:08:40.243916 1290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "aaea47adf0df4902bca5240d55f32349"
format_stamp: "Formatted at 2026-09-27 22:08:40 on dist-test-slave-mp03"
server_key: "653e5ccf314cbe19ce75e732ca3eb8d4"
server_key_iv: "09512bf2ff30fa6bb900defd80214d22"
server_key_version: "encryptionkey@0"
I20260927 22:08:40.244122 1290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:08:40.259665 1290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:08:40.264199 1290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:08:40.264519 1290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:08:40.265581 1290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:08:40.267040 1290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:08:40.267113 1290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:40.267292 1290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:08:40.267344 1290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:40.299641 1290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33157
I20260927 22:08:40.299682 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33157 every 8 connection(s)
I20260927 22:08:40.300984 1290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:08:40.305727 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 1290
I20260927 22:08:40.305913 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:08:40.306304 388 external_mini_cluster.cc:1428] Setting key 4f1476e51b669433e45fcd18e01492fe
Sep 27 22:08:40 dist-test-slave-mp03 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:08:40.321287 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:40.303487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43981 (local address 127.0.97.62:36829)
0927 22:08:40.303682 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:40.303688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:40.305045 (+ 1357us) server_negotiation.cc:407] Connection header received
0927 22:08:40.307047 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:40.307053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:40.307138 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:40.307252 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:40.309245 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.309990 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.311488 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.311886 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.311973 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:40.315490 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:40.315495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:40.315498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:40.315530 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:40.317939 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:40.318625 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:40.318631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:40.318635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:40.318753 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:40.319267 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:40.319270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:40.319272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:40.319596 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:40.319756 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:40.320656 (+ 900us) server_negotiation.cc:299] Negotiation successful
0927 22:08:40.320969 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:08:40.322245 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:40.304038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36829 (local address 127.0.97.3:43981)
0927 22:08:40.304887 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:40.304928 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:40.306730 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:40.307555 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:40.307566 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:40.308273 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:08:40.309074 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:40.309101 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.310113 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.310117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:40.311379 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:40.311389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.312055 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.312059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:40.312241 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:40.312906 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:08:40.312932 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:08:40.315284 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:08:40.318064 (+ 2780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:40.318071 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:40.318085 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:40.318493 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:40.318981 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:40.318985 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:40.318987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:40.319167 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:40.319744 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:08:40.319753 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:08:40.320180 (+ 427us) client_negotiation.cc:769] Sending connection context
0927 22:08:40.320509 (+ 329us) client_negotiation.cc:240] Negotiation successful
0927 22:08:40.321002 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":376,"threads_started":1}
I20260927 22:08:40.323588 1418 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36829
I20260927 22:08:40.323907 1418 heartbeater.cc:464] Registering TS with master...
I20260927 22:08:40.324627 1418 heartbeater.cc:511] Master 127.0.97.62:36829 requested a full tablet report, sending...
I20260927 22:08:40.325946 962 ts_manager.cc:194] Registered new tserver with Master: aaea47adf0df4902bca5240d55f32349 (127.0.97.3:33157)
I20260927 22:08:40.326809 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43981
I20260927 22:08:40.333990 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:08:40.352195 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:40.339909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39604 (local address 127.0.97.62:36829)
0927 22:08:40.340064 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:40.340068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:40.341005 (+ 937us) server_negotiation.cc:407] Connection header received
0927 22:08:40.341131 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:40.341134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:40.341189 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:40.341268 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:40.342565 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.343566 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.344723 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.345083 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.345169 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:40.347389 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:40.347394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:40.347397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:40.347427 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:40.350063 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:40.350678 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:40.350681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:40.350684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:40.350790 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:40.351085 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:40.351088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:40.351090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:40.351361 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:40.351537 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:40.351821 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 22:08:40.351971 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:08:40.363600 1151 heartbeater.cc:503] Master 127.0.97.62:36829 was elected leader, sending a full tablet report...
I20260927 22:08:40.358208 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:08:40.365681 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:08:40.402527 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:40.377689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56560 (local address 127.0.97.1:43287)
0927 22:08:40.390717 (+ 13028us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:40.390729 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:40.390763 (+ 34us) server_negotiation.cc:407] Connection header received
0927 22:08:40.390884 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:40.390892 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:40.391248 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:40.391536 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:08:40.392384 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.394502 (+ 2118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.398940 (+ 4438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.400887 (+ 1947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.401034 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:40.401310 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:40.401995 (+ 685us) server_negotiation.cc:299] Negotiation successful
0927 22:08:40.402174 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12818,"thread_start_us":12685,"threads_started":1}
I20260927 22:08:40.402626 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:40.377489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43287 (local address 127.0.0.1:56560)
0927 22:08:40.388398 (+ 10909us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:40.388443 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:40.388654 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:40.391567 (+ 2913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:40.391579 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:40.391630 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:08:40.392156 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:40.392175 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.394708 (+ 2533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.394715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:40.398656 (+ 3941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:40.398679 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.401453 (+ 2774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.401458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:40.401690 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:40.401759 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 22:08:40.401931 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 22:08:40.402166 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10618,"mutex_wait_us":3,"thread_start_us":10466,"threads_started":1}
I20260927 22:08:40.408007 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:40.378191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51068 (local address 127.0.97.2:38949)
0927 22:08:40.391056 (+ 12865us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:40.391064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:40.400492 (+ 9428us) server_negotiation.cc:407] Connection header received
0927 22:08:40.400836 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:40.400848 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:40.401113 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:40.401299 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:08:40.402153 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.403700 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.405991 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.406963 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.407130 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:40.407329 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:40.407520 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 22:08:40.407716 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12721,"thread_start_us":12590,"threads_started":1}
I20260927 22:08:40.407763 1086 tablet_service.cc:1467] Processing CreateTablet for tablet ecd1aff07af94d4c8c400d5322d4a2fb (DEFAULT_TABLE table=test-table [id=e893c448e02e43fbbe787bde289f61dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:08:40.408753 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:40.378202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38949 (local address 127.0.0.1:51068)
0927 22:08:40.400367 (+ 22165us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:40.400388 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:40.400598 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:40.401593 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:40.401599 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:40.401626 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:08:40.401987 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:40.401997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.403883 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.403888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:40.405844 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:40.405856 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.407160 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.407166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:40.407322 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:40.407359 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:08:40.408265 (+ 906us) client_negotiation.cc:240] Negotiation successful
0927 22:08:40.408398 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22053,"mutex_wait_us":1,"thread_start_us":20652,"threads_started":1}
I20260927 22:08:40.409444 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd1aff07af94d4c8c400d5322d4a2fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:40.410310 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:40.377525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33157 (local address 127.0.0.1:43488)
0927 22:08:40.399253 (+ 21728us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:40.399281 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:40.399442 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:40.401014 (+ 1572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:40.401019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:40.401040 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:08:40.401486 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:40.401501 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.403658 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.403667 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:40.406957 (+ 3290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:40.406973 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.409501 (+ 2528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.409507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:40.409696 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:40.409729 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:08:40.409829 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 22:08:40.409957 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21495,"mutex_wait_us":6,"thread_start_us":22568,"threads_started":1}
I20260927 22:08:40.410360 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:40.378114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43488 (local address 127.0.97.3:33157)
0927 22:08:40.400041 (+ 21927us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:40.400052 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:40.400093 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:08:40.400216 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:40.400223 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:40.400521 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:40.400747 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:08:40.401721 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.403483 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.407156 (+ 3673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.409299 (+ 2143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.409438 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:40.409665 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:40.409897 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 22:08:40.410045 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21753,"thread_start_us":21610,"threads_started":1}
I20260927 22:08:40.412587 1220 tablet_service.cc:1467] Processing CreateTablet for tablet ecd1aff07af94d4c8c400d5322d4a2fb (DEFAULT_TABLE table=test-table [id=e893c448e02e43fbbe787bde289f61dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:08:40.413985 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd1aff07af94d4c8c400d5322d4a2fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:40.414096 1353 tablet_service.cc:1467] Processing CreateTablet for tablet ecd1aff07af94d4c8c400d5322d4a2fb (DEFAULT_TABLE table=test-table [id=e893c448e02e43fbbe787bde289f61dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:08:40.415467 1353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd1aff07af94d4c8c400d5322d4a2fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:40.421371 1436 tablet_bootstrap.cc:492] T ecd1aff07af94d4c8c400d5322d4a2fb P 1c5480b12bc8496fb03fc7d87ba0db69: Bootstrap starting.
I20260927 22:08:40.424903 1436 tablet_bootstrap.cc:654] T ecd1aff07af94d4c8c400d5322d4a2fb P 1c5480b12bc8496fb03fc7d87ba0db69: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:40.425908 1436 log.cc:824] T ecd1aff07af94d4c8c400d5322d4a2fb P 1c5480b12bc8496fb03fc7d87ba0db69: Log is configured to *not* fsync() on all Append() calls
I20260927 22:08:40.428399 1436 tablet_bootstrap.cc:492] T ecd1aff07af94d4c8c400d5322d4a2fb P 1c5480b12bc8496fb03fc7d87ba0db69: No bootstrap required, opened a new log
I20260927 22:08:40.428721 1436 ts_tablet_manager.cc:1397] T ecd1aff07af94d4c8c400d5322d4a2fb P 1c5480b12bc8496fb03fc7d87ba0db69: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 22:08:40.429219 1438 tablet_bootstrap.cc:492] T ecd1aff07af94d4c8c400d5322d4a2fb P 7100e0b028c449bd8e2d48850ec6b012: Bootstrap starting.
I20260927 22:08:40.430581 1439 tablet_bootstrap.cc:492] T ecd1aff07af94d4c8c400d5322d4a2fb P aaea47adf0df4902bca5240d55f32349: Bootstrap starting.
I20260927 22:08:40.433892 1439 tablet_bootstrap.cc:654] T ecd1aff07af94d4c8c400d5322d4a2fb P aaea47adf0df4902bca5240d55f32349: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:40.434257 1438 tablet_bootstrap.cc:654] T ecd1aff07af94d4c8c400d5322d4a2fb P 7100e0b028c449bd8e2d48850ec6b012: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:40.434962 1439 log.cc:824] T ecd1aff07af94d4c8c400d5322d4a2fb P aaea47adf0df4902bca5240d55f32349: Log is configured to *not* fsync() on all Append() calls
I20260927 22:08:40.435428 1438 log.cc:824] T ecd1aff07af94d4c8c400d5322d4a2fb P 7100e0b028c449bd8e2d48850ec6b012: Log is configured to *not* fsync() on all Append() calls
I20260927 22:08:40.437585 1439 tablet_bootstrap.cc:492] T ecd1aff07af94d4c8c400d5322d4a2fb P aaea47adf0df4902bca5240d55f32349: No bootstrap required, opened a new log
I20260927 22:08:40.437844 1439 ts_tablet_manager.cc:1397] T ecd1aff07af94d4c8c400d5322d4a2fb P aaea47adf0df4902bca5240d55f32349: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260927 22:08:40.439985 1438 tablet_bootstrap.cc:492] T ecd1aff07af94d4c8c400d5322d4a2fb P 7100e0b028c449bd8e2d48850ec6b012: No bootstrap required, opened a new log
I20260927 22:08:40.439842 1436 raft_consensus.cc:348] T ecd1aff07af94d4c8c400d5322d4a2fb P 1c5480b12bc8496fb03fc7d87ba0db69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5480b12bc8496fb03fc7d87ba0db69" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43287 } } peers { permanent_uuid: "aaea47adf0df4902bca5240d55f32349" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33157 } } peers { permanent_uuid: "7100e0b028c449bd8e2d48850ec6b012" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38949 } }
I20260927 22:08:40.440239 1438 ts_tablet_manager.cc:1397] T ecd1aff07af94d4c8c400d5322d4a2fb P 7100e0b028c449bd8e2d48850ec6b012: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.006s
I20260927 22:08:40.440267 1436 raft_consensus.cc:374] T ecd1aff07af94d4c8c400d5322d4a2fb P 1c5480b12bc8496fb03fc7d87ba0db69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:40.440356 1436 raft_consensus.cc:729] T ecd1aff07af94d4c8c400d5322d4a2fb P 1c5480b12bc8496fb03fc7d87ba0db69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c5480b12bc8496fb03fc7d87ba0db69, State: Initialized, Role: FOLLOWER
I20260927 22:08:40.440999 1436 consensus_queue.cc:260] T ecd1aff07af94d4c8c400d5322d4a2fb P 1c5480b12bc8496fb03fc7d87ba0db69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5480b12bc8496fb03fc7d87ba0db69" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43287 } } peers { permanent_uuid: "aaea47adf0df4902bca5240d55f32349" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33157 } } peers { permanent_uuid: "7100e0b028c449bd8e2d48850ec6b012" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38949 } }
I20260927 22:08:40.442735 1436 ts_tablet_manager.cc:1428] T ecd1aff07af94d4c8c400d5322d4a2fb P 1c5480b12bc8496fb03fc7d87ba0db69: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260927 22:08:40.447019 1439 raft_consensus.cc:348] T ecd1aff07af94d4c8c400d5322d4a2fb P aaea47adf0df4902bca5240d55f32349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5480b12bc8496fb03fc7d87ba0db69" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43287 } } peers { permanent_uuid: "aaea47adf0df4902bca5240d55f32349" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33157 } } peers { permanent_uuid: "7100e0b028c449bd8e2d48850ec6b012" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38949 } }
I20260927 22:08:40.447475 1439 raft_consensus.cc:374] T ecd1aff07af94d4c8c400d5322d4a2fb P aaea47adf0df4902bca5240d55f32349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:40.447597 1439 raft_consensus.cc:729] T ecd1aff07af94d4c8c400d5322d4a2fb P aaea47adf0df4902bca5240d55f32349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaea47adf0df4902bca5240d55f32349, State: Initialized, Role: FOLLOWER
I20260927 22:08:40.448227 1439 consensus_queue.cc:260] T ecd1aff07af94d4c8c400d5322d4a2fb P aaea47adf0df4902bca5240d55f32349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5480b12bc8496fb03fc7d87ba0db69" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43287 } } peers { permanent_uuid: "aaea47adf0df4902bca5240d55f32349" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33157 } } peers { permanent_uuid: "7100e0b028c449bd8e2d48850ec6b012" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38949 } }
I20260927 22:08:40.449551 1418 heartbeater.cc:503] Master 127.0.97.62:36829 was elected leader, sending a full tablet report...
I20260927 22:08:40.450551 1438 raft_consensus.cc:348] T ecd1aff07af94d4c8c400d5322d4a2fb P 7100e0b028c449bd8e2d48850ec6b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5480b12bc8496fb03fc7d87ba0db69" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43287 } } peers { permanent_uuid: "aaea47adf0df4902bca5240d55f32349" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33157 } } peers { permanent_uuid: "7100e0b028c449bd8e2d48850ec6b012" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38949 } }
I20260927 22:08:40.450989 1438 raft_consensus.cc:374] T ecd1aff07af94d4c8c400d5322d4a2fb P 7100e0b028c449bd8e2d48850ec6b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:40.451107 1438 raft_consensus.cc:729] T ecd1aff07af94d4c8c400d5322d4a2fb P 7100e0b028c449bd8e2d48850ec6b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7100e0b028c449bd8e2d48850ec6b012, State: Initialized, Role: FOLLOWER
I20260927 22:08:40.451175 1439 ts_tablet_manager.cc:1428] T ecd1aff07af94d4c8c400d5322d4a2fb P aaea47adf0df4902bca5240d55f32349: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260927 22:08:40.451692 1438 consensus_queue.cc:260] T ecd1aff07af94d4c8c400d5322d4a2fb P 7100e0b028c449bd8e2d48850ec6b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5480b12bc8496fb03fc7d87ba0db69" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43287 } } peers { permanent_uuid: "aaea47adf0df4902bca5240d55f32349" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33157 } } peers { permanent_uuid: "7100e0b028c449bd8e2d48850ec6b012" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38949 } }
I20260927 22:08:40.452848 1285 heartbeater.cc:503] Master 127.0.97.62:36829 was elected leader, sending a full tablet report...
I20260927 22:08:40.453280 1438 ts_tablet_manager.cc:1428] T ecd1aff07af94d4c8c400d5322d4a2fb P 7100e0b028c449bd8e2d48850ec6b012: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
W20260927 22:08:40.552809 1419 tablet.cc:2369] T ecd1aff07af94d4c8c400d5322d4a2fb P aaea47adf0df4902bca5240d55f32349: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:08:40.565171 1286 tablet.cc:2369] T ecd1aff07af94d4c8c400d5322d4a2fb P 7100e0b028c449bd8e2d48850ec6b012: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:08:40.584281 1152 tablet.cc:2369] T ecd1aff07af94d4c8c400d5322d4a2fb P 1c5480b12bc8496fb03fc7d87ba0db69: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:08:40.694100 1443 raft_consensus.cc:482] T ecd1aff07af94d4c8c400d5322d4a2fb P aaea47adf0df4902bca5240d55f32349 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:08:40.694403 1443 raft_consensus.cc:504] T ecd1aff07af94d4c8c400d5322d4a2fb P aaea47adf0df4902bca5240d55f32349 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5480b12bc8496fb03fc7d87ba0db69" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43287 } } peers { permanent_uuid: "aaea47adf0df4902bca5240d55f32349" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33157 } } peers { permanent_uuid: "7100e0b028c449bd8e2d48850ec6b012" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38949 } }
I20260927 22:08:40.696115 1443 leader_election.cc:288] T ecd1aff07af94d4c8c400d5322d4a2fb P aaea47adf0df4902bca5240d55f32349 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c5480b12bc8496fb03fc7d87ba0db69 (127.0.97.1:43287), 7100e0b028c449bd8e2d48850ec6b012 (127.0.97.2:38949)
I20260927 22:08:40.703032 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:40.696528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60959 (local address 127.0.97.2:38949)
0927 22:08:40.696738 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:40.696746 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:40.696842 (+ 96us) server_negotiation.cc:407] Connection header received
0927 22:08:40.697061 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:40.697066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:40.697144 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:40.697253 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:08:40.697685 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.698962 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.701161 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.702201 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.702330 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:40.702438 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:40.702655 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 22:08:40.702778 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:08:40.703038 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:40.696398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38949 (local address 127.0.97.3:60959)
0927 22:08:40.696751 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:40.696769 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:40.696894 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:40.697288 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:40.697293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:40.697323 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:08:40.697537 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:40.697550 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.699089 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.699093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:40.701005 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:40.701019 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.702354 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.702357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:40.702498 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:40.702521 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:08:40.702588 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 22:08:40.702746 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":23}
I20260927 22:08:40.703752 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:40.696562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43287 (local address 127.0.97.3:36181)
0927 22:08:40.697324 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:40.697340 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:40.697447 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:40.697751 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:40.697753 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:40.697771 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:08:40.698430 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:40.698438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.699575 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.699581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:40.701409 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:40.701419 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.702980 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.702985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:40.703165 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:40.703201 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:08:40.703326 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 22:08:40.703445 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":1,"thread_start_us":444,"threads_started":1}
I20260927 22:08:40.703748 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:40.696468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36181 (local address 127.0.97.1:43287)
0927 22:08:40.696672 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:40.696678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:40.697428 (+ 750us) server_negotiation.cc:407] Connection header received
0927 22:08:40.697558 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:40.697562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:40.697641 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:40.697736 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:08:40.698573 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.699439 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.701545 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.702777 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.702919 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:40.703011 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:40.703371 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 22:08:40.703503 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:08:40.704082 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd1aff07af94d4c8c400d5322d4a2fb" candidate_uuid: "aaea47adf0df4902bca5240d55f32349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7100e0b028c449bd8e2d48850ec6b012" is_pre_election: true
I20260927 22:08:40.704666 1240 raft_consensus.cc:2454] T ecd1aff07af94d4c8c400d5322d4a2fb P 7100e0b028c449bd8e2d48850ec6b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaea47adf0df4902bca5240d55f32349 in term 0.
I20260927 22:08:40.704797 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd1aff07af94d4c8c400d5322d4a2fb" candidate_uuid: "aaea47adf0df4902bca5240d55f32349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c5480b12bc8496fb03fc7d87ba0db69" is_pre_election: true
I20260927 22:08:40.705283 1106 raft_consensus.cc:2454] T ecd1aff07af94d4c8c400d5322d4a2fb P 1c5480b12bc8496fb03fc7d87ba0db69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaea47adf0df4902bca5240d55f32349 in term 0.
I20260927 22:08:40.705334 1306 leader_election.cc:302] T ecd1aff07af94d4c8c400d5322d4a2fb P aaea47adf0df4902bca5240d55f32349 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7100e0b028c449bd8e2d48850ec6b012, aaea47adf0df4902bca5240d55f32349; no voters:
I20260927 22:08:40.705680 1443 raft_consensus.cc:2781] T ecd1aff07af94d4c8c400d5322d4a2fb P aaea47adf0df4902bca5240d55f32349 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:08:40.705854 1443 raft_consensus.cc:482] T ecd1aff07af94d4c8c400d5322d4a2fb P aaea47adf0df4902bca5240d55f32349 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:08:40.705972 1443 raft_consensus.cc:3037] T ecd1aff07af94d4c8c400d5322d4a2fb P aaea47adf0df4902bca5240d55f32349 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:40.708122 1443 raft_consensus.cc:504] T ecd1aff07af94d4c8c400d5322d4a2fb P aaea47adf0df4902bca5240d55f32349 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5480b12bc8496fb03fc7d87ba0db69" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43287 } } peers { permanent_uuid: "aaea47adf0df4902bca5240d55f32349" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33157 } } peers { permanent_uuid: "7100e0b028c449bd8e2d48850ec6b012" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38949 } }
I20260927 22:08:40.709160 1443 leader_election.cc:288] T ecd1aff07af94d4c8c400d5322d4a2fb P aaea47adf0df4902bca5240d55f32349 [CANDIDATE]: Term 1 election: Requested vote from peers 1c5480b12bc8496fb03fc7d87ba0db69 (127.0.97.1:43287), 7100e0b028c449bd8e2d48850ec6b012 (127.0.97.2:38949)
I20260927 22:08:40.709494 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd1aff07af94d4c8c400d5322d4a2fb" candidate_uuid: "aaea47adf0df4902bca5240d55f32349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c5480b12bc8496fb03fc7d87ba0db69"
I20260927 22:08:40.709609 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd1aff07af94d4c8c400d5322d4a2fb" candidate_uuid: "aaea47adf0df4902bca5240d55f32349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7100e0b028c449bd8e2d48850ec6b012"
I20260927 22:08:40.709687 1106 raft_consensus.cc:3037] T ecd1aff07af94d4c8c400d5322d4a2fb P 1c5480b12bc8496fb03fc7d87ba0db69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:40.709795 1240 raft_consensus.cc:3037] T ecd1aff07af94d4c8c400d5322d4a2fb P 7100e0b028c449bd8e2d48850ec6b012 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:40.712862 1106 raft_consensus.cc:2454] T ecd1aff07af94d4c8c400d5322d4a2fb P 1c5480b12bc8496fb03fc7d87ba0db69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaea47adf0df4902bca5240d55f32349 in term 1.
I20260927 22:08:40.712886 1240 raft_consensus.cc:2454] T ecd1aff07af94d4c8c400d5322d4a2fb P 7100e0b028c449bd8e2d48850ec6b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaea47adf0df4902bca5240d55f32349 in term 1.
I20260927 22:08:40.713227 1308 leader_election.cc:302] T ecd1aff07af94d4c8c400d5322d4a2fb P aaea47adf0df4902bca5240d55f32349 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c5480b12bc8496fb03fc7d87ba0db69, aaea47adf0df4902bca5240d55f32349; no voters:
I20260927 22:08:40.713506 1443 raft_consensus.cc:2781] T ecd1aff07af94d4c8c400d5322d4a2fb P aaea47adf0df4902bca5240d55f32349 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:08:40.713996 1443 raft_consensus.cc:686] T ecd1aff07af94d4c8c400d5322d4a2fb P aaea47adf0df4902bca5240d55f32349 [term 1 LEADER]: Becoming Leader. State: Replica: aaea47adf0df4902bca5240d55f32349, State: Running, Role: LEADER
I20260927 22:08:40.714895 1443 consensus_queue.cc:237] T ecd1aff07af94d4c8c400d5322d4a2fb P aaea47adf0df4902bca5240d55f32349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5480b12bc8496fb03fc7d87ba0db69" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43287 } } peers { permanent_uuid: "aaea47adf0df4902bca5240d55f32349" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33157 } } peers { permanent_uuid: "7100e0b028c449bd8e2d48850ec6b012" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38949 } }
I20260927 22:08:40.719345 961 catalog_manager.cc:5495] T ecd1aff07af94d4c8c400d5322d4a2fb P aaea47adf0df4902bca5240d55f32349 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaea47adf0df4902bca5240d55f32349 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "aaea47adf0df4902bca5240d55f32349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c5480b12bc8496fb03fc7d87ba0db69" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaea47adf0df4902bca5240d55f32349" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7100e0b028c449bd8e2d48850ec6b012" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38949 } health_report { overall_health: UNKNOWN } } }
Sep 27 22:08:40 dist-test-slave-mp03 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:08:40.780342 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:40.765934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56572 (local address 127.0.97.1:43287)
0927 22:08:40.766107 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:40.766111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:40.766670 (+ 559us) server_negotiation.cc:407] Connection header received
0927 22:08:40.766884 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:40.766888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:40.766966 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:40.767074 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:40.768552 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.769274 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.770496 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.771003 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.771137 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:40.775097 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:40.775117 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:40.775134 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:40.775169 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:40.777733 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:40.778357 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:40.778366 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:40.778380 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:40.778506 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:40.778875 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:40.778882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:40.778888 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:40.779225 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:40.779527 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:40.779795 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 22:08:40.780003 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 22:08:40 dist-test-slave-mp03 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546920, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:08:40 dist-test-slave-mp03 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546920, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:08:40.826185 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:40.810643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56586 (local address 127.0.97.1:43287)
0927 22:08:40.810870 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:40.810877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:40.811233 (+ 356us) server_negotiation.cc:407] Connection header received
0927 22:08:40.811430 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:40.811452 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:40.811534 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:40.811618 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:40.813250 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.813962 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.815774 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.816294 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.816387 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:40.820765 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:40.820770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:40.820773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:40.820804 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:40.823780 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:40.824482 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:40.824486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:40.824488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:40.824606 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:40.824996 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:40.824999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:40.825001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:40.825298 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:40.825506 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:40.825776 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 22:08:40.825928 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 22:08:40 dist-test-slave-mp03 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:08:40 dist-test-slave-mp03 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:08:40.868335 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:40.855424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56602 (local address 127.0.97.1:43287)
0927 22:08:40.855674 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:40.855680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:40.855843 (+ 163us) server_negotiation.cc:407] Connection header received
0927 22:08:40.855991 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:40.855994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:40.856055 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:40.856138 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:40.857335 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.857990 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.859213 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.859621 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.859712 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:40.863460 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:40.863465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:40.863468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:40.863497 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:40.866046 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:40.866734 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:40.866738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:40.866740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:40.866848 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:40.867149 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:40.867152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:40.867154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:40.867415 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:40.867593 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:40.867857 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 22:08:40.868036 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260927 22:08:40.869959 1066 tablet_service.cc:2908] Rejecting scan request for tablet ecd1aff07af94d4c8c400d5322d4a2fb: Uninitialized: clean time has not yet been initialized
I20260927 22:08:40.873250 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 1023
I20260927 22:08:40.885375 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:40.874126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56606 (local address 127.0.97.1:43287)
0927 22:08:40.874295 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:40.874299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:40.874681 (+ 382us) server_negotiation.cc:407] Connection header received
0927 22:08:40.874832 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:40.874836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:40.874905 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:40.874982 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:40.876120 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.876763 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.877926 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.878341 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.878454 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:40.881031 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:40.881035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:40.881038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:40.881074 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:40.883333 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:40.883870 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:40.883874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:40.883876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:40.883979 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:40.884328 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:40.884331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:40.884333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:40.884610 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:40.884801 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:40.885011 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 22:08:40.885152 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:08:40.886009 1146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:08:40.973476 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 1023
I20260927 22:08:40.990684 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 1157
Sep 27 22:08:40 dist-test-slave-mp03 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:08:41.004942 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:40.991435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51072 (local address 127.0.97.2:38949)
0927 22:08:40.991621 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:40.991626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:40.991647 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:08:40.991774 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:40.991778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:40.991843 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:40.991949 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:40.993059 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.993749 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.995054 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:40.995427 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:40.995560 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:40.999418 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:40.999432 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:40.999448 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:40.999483 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:41.002254 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:41.002989 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:41.002996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:41.002998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:41.003120 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:41.003490 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:41.003495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:41.003505 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:41.003846 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:41.004158 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:41.004378 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 22:08:41.004599 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:08:41.005548 1280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:08:41.094489 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 1157
I20260927 22:08:41.110929 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 1290
Sep 27 22:08:41 dist-test-slave-mp03 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:08:41.125835 1461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:41.111744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43504 (local address 127.0.97.3:33157)
0927 22:08:41.112800 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:41.112806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:41.112825 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:08:41.112888 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:41.112893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:41.112967 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:41.113105 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:41.114521 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:41.115264 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:41.116477 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:41.116884 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:41.117018 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:41.121005 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:41.121019 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:41.121037 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:41.121069 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:41.123624 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:41.124228 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:41.124235 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:41.124237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:41.124357 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:41.124668 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:41.124671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:41.124673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:41.125001 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:41.125203 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:41.125395 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 22:08:41.125553 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":929,"thread_start_us":345,"threads_started":1}
I20260927 22:08:41.126442 1413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:08:41.217801 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 1290
I20260927 22:08:41.234332 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 930
I20260927 22:08:41.235605 992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:08:41.341245 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 930
2026-09-27T22:08:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4307 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:08:41 dist-test-slave-mp03 krb5kdc[1467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:08:41 dist-test-slave-mp03 krb5kdc[1467](info): set up 2 sockets
Sep 27 22:08:41 dist-test-slave-mp03 krb5kdc[1467](info): commencing operation
krb5kdc: starting...
W20260927 22:08:42.418550 388 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.004s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:08:42 dist-test-slave-mp03 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:08:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:08:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:08:42.598408 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33057
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:08:42.916118 1480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:08:42.916399 1480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:08:42.916571 1480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:08:42.923755 1480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:08:42.923851 1480 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 22:08:42.923939 1480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:08:42.924043 1480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:08:42.924137 1480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:08:42.924248 1480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:08:42.934300 1480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33057
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:08:42.935629 1480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:08:42.936964 1480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:08:42.950066 1485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:42.950035 1486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:42.950099 1488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:08:42.950896 1480 server_base.cc:1034] running on GCE node
I20260927 22:08:42.951699 1480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:08:42.953150 1480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:08:42.954403 1480 hybrid_clock.cc:648] HybridClock initialized: now 1790546922954339 us; error 43 us; skew 500 ppm
I20260927 22:08:42.954795 1480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:08:42 dist-test-slave-mp03 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:08:42.957805 1480 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:08:42.959342 1480 webserver.cc:460] Webserver started at http://127.0.97.62:38191/ using document root <none> and password file <none>
I20260927 22:08:42.960063 1480 fs_manager.cc:359] Metadata directory not provided
I20260927 22:08:42.960198 1480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:08:42.960529 1480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:08:42.963533 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "b949e81c6e354c868629b02e622250ce"
format_stamp: "Formatted at 2026-09-27 22:08:42 on dist-test-slave-mp03"
server_key: "14a1a37766f31053a18451dad1d42403"
server_key_iv: "4111a67ee5267d79c1ecee81ea574585"
server_key_version: "encryptionkey@0"
I20260927 22:08:42.964361 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "b949e81c6e354c868629b02e622250ce"
format_stamp: "Formatted at 2026-09-27 22:08:42 on dist-test-slave-mp03"
server_key: "14a1a37766f31053a18451dad1d42403"
server_key_iv: "4111a67ee5267d79c1ecee81ea574585"
server_key_version: "encryptionkey@0"
I20260927 22:08:42.969532 1480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:08:42.972988 1495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:42.974324 1480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:08:42.974519 1480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "b949e81c6e354c868629b02e622250ce"
format_stamp: "Formatted at 2026-09-27 22:08:42 on dist-test-slave-mp03"
server_key: "14a1a37766f31053a18451dad1d42403"
server_key_iv: "4111a67ee5267d79c1ecee81ea574585"
server_key_version: "encryptionkey@0"
I20260927 22:08:42.974850 1480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:08:42.996570 1480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:08:43.000686 1480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:08:43.000955 1480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:08:43.022281 1480 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33057
I20260927 22:08:43.022303 1556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33057 every 8 connection(s)
I20260927 22:08:43.023710 1480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:08:43.025580 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 1480
I20260927 22:08:43.025806 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:08:43.026615 388 external_mini_cluster.cc:1428] Setting key 3e8b895d4cd93a798bae7bf0fbfe0e29
I20260927 22:08:43.028469 1557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:08:43 dist-test-slave-mp03 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:08:43.041987 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b949e81c6e354c868629b02e622250ce: Bootstrap starting.
I20260927 22:08:43.045256 1557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b949e81c6e354c868629b02e622250ce: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:43.046274 1557 log.cc:824] T 00000000000000000000000000000000 P b949e81c6e354c868629b02e622250ce: Log is configured to *not* fsync() on all Append() calls
I20260927 22:08:43.049178 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b949e81c6e354c868629b02e622250ce: No bootstrap required, opened a new log
I20260927 22:08:43.049757 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:43.028709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55670 (local address 127.0.97.62:33057)
0927 22:08:43.029443 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:43.029454 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:43.029494 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:08:43.031102 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:43.031123 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:43.031510 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:43.031878 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:43.033548 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:43.035015 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:43.036757 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:43.037506 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:43.037663 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:43.042365 (+ 4702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:43.042373 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:43.042388 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:43.042426 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:43.045696 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:43.046376 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:43.046388 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:43.046413 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:43.046586 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:43.047005 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:43.047012 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:43.047015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:43.047565 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:43.047871 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:43.048289 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 22:08:43.048701 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":346,"threads_started":1}
I20260927 22:08:43.056981 1557 raft_consensus.cc:348] T 00000000000000000000000000000000 P b949e81c6e354c868629b02e622250ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b949e81c6e354c868629b02e622250ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33057 } }
I20260927 22:08:43.057348 1557 raft_consensus.cc:374] T 00000000000000000000000000000000 P b949e81c6e354c868629b02e622250ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:43.057456 1557 raft_consensus.cc:729] T 00000000000000000000000000000000 P b949e81c6e354c868629b02e622250ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b949e81c6e354c868629b02e622250ce, State: Initialized, Role: FOLLOWER
I20260927 22:08:43.058022 1557 consensus_queue.cc:260] T 00000000000000000000000000000000 P b949e81c6e354c868629b02e622250ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b949e81c6e354c868629b02e622250ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33057 } }
I20260927 22:08:43.058255 1557 raft_consensus.cc:388] T 00000000000000000000000000000000 P b949e81c6e354c868629b02e622250ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:08:43.058394 1557 raft_consensus.cc:482] T 00000000000000000000000000000000 P b949e81c6e354c868629b02e622250ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:08:43.058561 1557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b949e81c6e354c868629b02e622250ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:43.060961 1557 raft_consensus.cc:504] T 00000000000000000000000000000000 P b949e81c6e354c868629b02e622250ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b949e81c6e354c868629b02e622250ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33057 } }
I20260927 22:08:43.061436 1557 leader_election.cc:302] T 00000000000000000000000000000000 P b949e81c6e354c868629b02e622250ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b949e81c6e354c868629b02e622250ce; no voters:
I20260927 22:08:43.061969 1557 leader_election.cc:288] T 00000000000000000000000000000000 P b949e81c6e354c868629b02e622250ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:08:43.062103 1562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b949e81c6e354c868629b02e622250ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:08:43.062825 1562 raft_consensus.cc:686] T 00000000000000000000000000000000 P b949e81c6e354c868629b02e622250ce [term 1 LEADER]: Becoming Leader. State: Replica: b949e81c6e354c868629b02e622250ce, State: Running, Role: LEADER
I20260927 22:08:43.063431 1562 consensus_queue.cc:237] T 00000000000000000000000000000000 P b949e81c6e354c868629b02e622250ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b949e81c6e354c868629b02e622250ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33057 } }
I20260927 22:08:43.064239 1557 sys_catalog.cc:564] T 00000000000000000000000000000000 P b949e81c6e354c868629b02e622250ce [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:08:43.066263 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P b949e81c6e354c868629b02e622250ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader b949e81c6e354c868629b02e622250ce. Latest consensus state: current_term: 1 leader_uuid: "b949e81c6e354c868629b02e622250ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b949e81c6e354c868629b02e622250ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33057 } } }
I20260927 22:08:43.066615 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P b949e81c6e354c868629b02e622250ce [sys.catalog]: This master's current role is: LEADER
I20260927 22:08:43.067600 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P b949e81c6e354c868629b02e622250ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b949e81c6e354c868629b02e622250ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b949e81c6e354c868629b02e622250ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33057 } } }
I20260927 22:08:43.068035 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P b949e81c6e354c868629b02e622250ce [sys.catalog]: This master's current role is: LEADER
I20260927 22:08:43.069401 1569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:08:43.075327 1569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:08:43.085382 1569 catalog_manager.cc:1347] Generated new cluster ID: a6b76b2dcda34d61a0b041a2795538ef
I20260927 22:08:43.085492 1569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:08:43.124922 1569 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:08:43.126255 1569 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:08:43.134904 1569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b949e81c6e354c868629b02e622250ce: Generated new TSK 0
I20260927 22:08:43.135735 1569 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:08:43.198274 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33057
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:08:43.516407 1583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:08:43.516683 1583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:08:43.516855 1583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:08:43.524045 1583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:08:43.524148 1583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:08:43.524344 1583 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:08:43.524432 1583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:08:43.534142 1583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:08:43.535488 1583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:08:43.536840 1583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:08:43.550530 1588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:43.550578 1591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:43.550557 1589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:08:43.551208 1583 server_base.cc:1034] running on GCE node
I20260927 22:08:43.551837 1583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:08:43.552673 1583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:08:43.553915 1583 hybrid_clock.cc:648] HybridClock initialized: now 1790546923553863 us; error 56 us; skew 500 ppm
I20260927 22:08:43.554207 1583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:08:43 dist-test-slave-mp03 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:08:43.557199 1583 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:08:43.558673 1583 webserver.cc:460] Webserver started at http://127.0.97.1:44853/ using document root <none> and password file <none>
I20260927 22:08:43.559311 1583 fs_manager.cc:359] Metadata directory not provided
I20260927 22:08:43.559392 1583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:08:43.559665 1583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:08:43.562422 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "a8f11e96ea164f2f8d8792ad228d889b"
format_stamp: "Formatted at 2026-09-27 22:08:43 on dist-test-slave-mp03"
server_key: "054e375acccd6fa0f76bc8d27ad13c55"
server_key_iv: "30823de7d4f8ab47e99c69d7d021e18d"
server_key_version: "encryptionkey@0"
I20260927 22:08:43.563197 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a8f11e96ea164f2f8d8792ad228d889b"
format_stamp: "Formatted at 2026-09-27 22:08:43 on dist-test-slave-mp03"
server_key: "054e375acccd6fa0f76bc8d27ad13c55"
server_key_iv: "30823de7d4f8ab47e99c69d7d021e18d"
server_key_version: "encryptionkey@0"
I20260927 22:08:43.568397 1583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:08:43.572036 1598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:43.573361 1583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:08:43.573491 1583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "a8f11e96ea164f2f8d8792ad228d889b"
format_stamp: "Formatted at 2026-09-27 22:08:43 on dist-test-slave-mp03"
server_key: "054e375acccd6fa0f76bc8d27ad13c55"
server_key_iv: "30823de7d4f8ab47e99c69d7d021e18d"
server_key_version: "encryptionkey@0"
I20260927 22:08:43.573668 1583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:08:43.586289 1583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:08:43.590178 1583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:08:43.590406 1583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:08:43.592962 1583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:08:43.593092 1583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:43.593261 1583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:08:43.593339 1583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:08:43 dist-test-slave-mp03 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:08:43.615295 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:43.593336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45443 (local address 127.0.97.62:33057)
0927 22:08:43.593873 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:43.593880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:43.595002 (+ 1122us) server_negotiation.cc:407] Connection header received
0927 22:08:43.596954 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:43.596960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:43.597046 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:43.597161 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:43.599348 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:43.600396 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:43.602909 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:43.603457 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:43.603587 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:43.608265 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:43.608273 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:43.608286 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:43.608335 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:43.611449 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:43.612538 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:43.612542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:43.612546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:43.612698 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:43.613300 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:43.613305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:43.613311 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:43.613739 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:43.613919 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:43.614689 (+ 770us) server_negotiation.cc:299] Negotiation successful
0927 22:08:43.614921 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":248,"threads_started":1}
I20260927 22:08:43.616485 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:43.593971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33057 (local address 127.0.97.1:45443)
0927 22:08:43.594834 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:43.594883 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:43.596661 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:43.597392 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:43.597400 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:43.598176 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:08:43.599150 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:43.599180 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:43.600544 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:43.600553 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:43.602750 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:43.602762 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:43.603741 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:43.603750 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:43.603960 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:43.604645 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:08:43.604666 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:08:43.607945 (+ 3279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:08:43.611611 (+ 3666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:43.611620 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:43.611638 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:43.612378 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:43.612866 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:43.612873 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:43.612879 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:43.613171 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:43.613887 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:08:43.613895 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:08:43.614304 (+ 409us) client_negotiation.cc:769] Sending connection context
0927 22:08:43.614573 (+ 269us) client_negotiation.cc:240] Negotiation successful
0927 22:08:43.615154 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":358,"threads_started":1}
I20260927 22:08:43.638885 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:43.623143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47069 (local address 127.0.97.62:33057)
0927 22:08:43.623339 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:43.623348 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:43.624025 (+ 677us) server_negotiation.cc:407] Connection header received
0927 22:08:43.624211 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:43.624215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:43.624273 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:43.624352 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:43.625818 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:43.626493 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:43.629092 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:43.629919 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:43.630035 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:43.633012 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:43.633019 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:43.633025 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:43.633067 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:43.636399 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:43.637038 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:43.637045 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:43.637048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:43.637176 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:43.637507 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:43.637514 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:43.637518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:43.637763 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:43.637910 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:43.638286 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 22:08:43.638501 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:08:43.639037 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:43.623149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33057 (local address 127.0.97.1:47069)
0927 22:08:43.623905 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:43.623930 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:43.624097 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:43.624416 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:43.624424 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:43.625202 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:08:43.625655 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:43.625671 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:43.626640 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:43.626648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:43.628887 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:43.628907 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:43.630384 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:43.630397 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:43.630662 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:43.631541 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:08:43.631571 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:08:43.632758 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:08:43.636522 (+ 3764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:43.636528 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:43.636532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:43.636928 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:43.637280 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:43.637286 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:43.637290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:43.637408 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:43.637885 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:08:43.637889 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:08:43.638057 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 22:08:43.638197 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 22:08:43.638501 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":398,"threads_started":1}
I20260927 22:08:43.647959 1583 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34847
I20260927 22:08:43.647989 1718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34847 every 8 connection(s)
I20260927 22:08:43.649161 1583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:08:43.651119 1719 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33057
I20260927 22:08:43.651297 1719 heartbeater.cc:464] Registering TS with master...
I20260927 22:08:43.651849 1719 heartbeater.cc:511] Master 127.0.97.62:33057 requested a full tablet report, sending...
I20260927 22:08:43.653573 1512 ts_manager.cc:194] Registered new tserver with Master: a8f11e96ea164f2f8d8792ad228d889b (127.0.97.1:34847)
I20260927 22:08:43.654979 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45443
I20260927 22:08:43.655726 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 1583
I20260927 22:08:43.655952 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:08:43.656530 388 external_mini_cluster.cc:1428] Setting key 2f641d70e6e7458add41e2f850fb167f
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:08:43.694521 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33057
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:08:44.014631 1723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:08:44.014911 1723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:08:44.015004 1723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:08:44.022231 1723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:08:44.022341 1723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:08:44.022529 1723 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:08:44.022670 1723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:08:44.032539 1723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:08:44.033804 1723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:08:44.035174 1723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:08:44.048305 1731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:44.048345 1729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:44.048261 1728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:08:44.049641 1723 server_base.cc:1034] running on GCE node
I20260927 22:08:44.050246 1723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:08:44.051218 1723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:08:44.052456 1723 hybrid_clock.cc:648] HybridClock initialized: now 1790546924052393 us; error 43 us; skew 500 ppm
I20260927 22:08:44.052749 1723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:08:44 dist-test-slave-mp03 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:08:44.062129 1723 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:08:44.063665 1723 webserver.cc:460] Webserver started at http://127.0.97.2:46043/ using document root <none> and password file <none>
I20260927 22:08:44.064441 1723 fs_manager.cc:359] Metadata directory not provided
I20260927 22:08:44.064570 1723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:08:44.064888 1723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:08:44.067832 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "87daa5631c7748caa3118088bbcb7926"
format_stamp: "Formatted at 2026-09-27 22:08:44 on dist-test-slave-mp03"
server_key: "62de9fe2760cb12f6554816217015492"
server_key_iv: "5779e93b823cd5c3995ced98c9d57640"
server_key_version: "encryptionkey@0"
I20260927 22:08:44.068620 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "87daa5631c7748caa3118088bbcb7926"
format_stamp: "Formatted at 2026-09-27 22:08:44 on dist-test-slave-mp03"
server_key: "62de9fe2760cb12f6554816217015492"
server_key_iv: "5779e93b823cd5c3995ced98c9d57640"
server_key_version: "encryptionkey@0"
I20260927 22:08:44.073848 1723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:08:44.077379 1738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:44.078943 1723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:08:44.079113 1723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "87daa5631c7748caa3118088bbcb7926"
format_stamp: "Formatted at 2026-09-27 22:08:44 on dist-test-slave-mp03"
server_key: "62de9fe2760cb12f6554816217015492"
server_key_iv: "5779e93b823cd5c3995ced98c9d57640"
server_key_version: "encryptionkey@0"
I20260927 22:08:44.079377 1723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:08:44.118095 1723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:08:44.122318 1723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:08:44.122677 1723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:08:44.126080 1723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:08:44.126161 1723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:44.126252 1723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:08:44.126299 1723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:08:44 dist-test-slave-mp03 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:08:44.152204 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:44.126809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59429 (local address 127.0.97.62:33057)
0927 22:08:44.126999 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:44.127007 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:44.128399 (+ 1392us) server_negotiation.cc:407] Connection header received
0927 22:08:44.131647 (+ 3248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:44.131654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:44.131746 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:44.131872 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:44.133856 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.134650 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.137319 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.138237 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.138409 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:44.143992 (+ 5583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:44.144001 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:44.144008 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:44.144053 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:44.148220 (+ 4167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:44.149437 (+ 1217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:44.149446 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:44.149452 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:44.149600 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:44.150179 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:44.150187 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:44.150191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:44.150626 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:44.150817 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:44.151633 (+ 816us) server_negotiation.cc:299] Negotiation successful
0927 22:08:44.151855 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:08:44.153209 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:44.127333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33057 (local address 127.0.97.2:59429)
0927 22:08:44.128198 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:44.128253 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:44.131236 (+ 2983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:44.132157 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:44.132172 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:44.133076 (+ 904us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:08:44.133677 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:44.133694 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.134802 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.134807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:44.137150 (+ 2343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:44.137168 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.138457 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.138462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:44.138735 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:44.139615 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:08:44.139654 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:08:44.143624 (+ 3970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:08:44.148432 (+ 4808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:44.148450 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:44.148473 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:44.149270 (+ 797us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:44.149744 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:44.149752 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:44.149758 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:44.150043 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:44.150789 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:08:44.150797 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:08:44.151224 (+ 427us) client_negotiation.cc:769] Sending connection context
0927 22:08:44.151535 (+ 311us) client_negotiation.cc:240] Negotiation successful
0927 22:08:44.152065 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":323,"threads_started":1}
I20260927 22:08:44.173542 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:44.159139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59045 (local address 127.0.97.62:33057)
0927 22:08:44.159370 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:44.159378 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:44.159901 (+ 523us) server_negotiation.cc:407] Connection header received
0927 22:08:44.160040 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:44.160048 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:44.160151 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:44.160279 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:44.161174 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.162694 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.163969 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.164785 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.164949 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:44.167455 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:44.167461 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:44.167466 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:44.167527 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:44.170820 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:44.171655 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:44.171662 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:44.171665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:44.171780 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:44.172131 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:44.172137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:44.172141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:44.172462 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:44.172609 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:44.172991 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 22:08:44.173198 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:08:44.173688 1818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:44.159140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33057 (local address 127.0.97.2:59045)
0927 22:08:44.159806 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:44.159822 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:44.159930 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:44.160289 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:44.160292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:44.160711 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:08:44.161038 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:44.161046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.162838 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.162842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:44.163838 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:44.163849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.165013 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.165021 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:44.165171 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:44.165862 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:08:44.165883 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:08:44.167265 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:08:44.170944 (+ 3679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:44.170949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:44.170954 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:44.171558 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:44.171895 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:44.171900 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:44.171904 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:44.172028 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:44.172575 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:08:44.172580 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:08:44.172759 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 22:08:44.172907 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 22:08:44.173218 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":364,"threads_started":1}
I20260927 22:08:44.184628 1723 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41607
I20260927 22:08:44.184666 1857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41607 every 8 connection(s)
I20260927 22:08:44.185868 1723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:08:44.187888 1858 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33057
I20260927 22:08:44.188120 1858 heartbeater.cc:464] Registering TS with master...
I20260927 22:08:44.188758 1858 heartbeater.cc:511] Master 127.0.97.62:33057 requested a full tablet report, sending...
I20260927 22:08:44.190049 1512 ts_manager.cc:194] Registered new tserver with Master: 87daa5631c7748caa3118088bbcb7926 (127.0.97.2:41607)
I20260927 22:08:44.190910 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59429
I20260927 22:08:44.192135 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 1723
I20260927 22:08:44.192309 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:08:44.192721 388 external_mini_cluster.cc:1428] Setting key 48f4b5c85c269b054f7eab483d2b7eb8
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:08:44.230542 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33057
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:08:44.558799 1862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:08:44.559087 1862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:08:44.559319 1862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:08:44.566977 1862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:08:44.567104 1862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:08:44.567256 1862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:08:44.567387 1862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:08:44.577525 1862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:08:44.578940 1862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:08:44.580327 1862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:08:44.594959 1867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:44.594890 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:44.594883 1868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:08:44.595804 1862 server_base.cc:1034] running on GCE node
I20260927 22:08:44.596467 1862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:08:44.597445 1862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:08:44.598704 1862 hybrid_clock.cc:648] HybridClock initialized: now 1790546924598634 us; error 37 us; skew 500 ppm
I20260927 22:08:44.599023 1862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:08:44 dist-test-slave-mp03 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:08:44.602205 1862 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:08:44.603643 1862 webserver.cc:460] Webserver started at http://127.0.97.3:37735/ using document root <none> and password file <none>
I20260927 22:08:44.604300 1862 fs_manager.cc:359] Metadata directory not provided
I20260927 22:08:44.604388 1862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:08:44.604646 1862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:08:44.607540 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "e716f1aef97e4c6ea2d5bf5974390f1c"
format_stamp: "Formatted at 2026-09-27 22:08:44 on dist-test-slave-mp03"
server_key: "b06abe7418ac2aac48bef610a7d8b28a"
server_key_iv: "8cce0a1af2b942713823d79f083d45ef"
server_key_version: "encryptionkey@0"
I20260927 22:08:44.608258 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e716f1aef97e4c6ea2d5bf5974390f1c"
format_stamp: "Formatted at 2026-09-27 22:08:44 on dist-test-slave-mp03"
server_key: "b06abe7418ac2aac48bef610a7d8b28a"
server_key_iv: "8cce0a1af2b942713823d79f083d45ef"
server_key_version: "encryptionkey@0"
I20260927 22:08:44.613516 1862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:08:44.616945 1877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:44.618292 1862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:08:44.618548 1862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "e716f1aef97e4c6ea2d5bf5974390f1c"
format_stamp: "Formatted at 2026-09-27 22:08:44 on dist-test-slave-mp03"
server_key: "b06abe7418ac2aac48bef610a7d8b28a"
server_key_iv: "8cce0a1af2b942713823d79f083d45ef"
server_key_version: "encryptionkey@0"
I20260927 22:08:44.618839 1862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:08:44.653976 1862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:08:44.657837 1719 heartbeater.cc:503] Master 127.0.97.62:33057 was elected leader, sending a full tablet report...
I20260927 22:08:44.659173 1862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:08:44.659464 1862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:08:44.662176 1862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:08:44.662284 1862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:44.662420 1862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:08:44.662487 1862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:08:44 dist-test-slave-mp03 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:08:44.699344 1862 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37207
I20260927 22:08:44.699494 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:44.662963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38803 (local address 127.0.97.62:33057)
0927 22:08:44.663139 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:44.663150 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:44.665036 (+ 1886us) server_negotiation.cc:407] Connection header received
0927 22:08:44.667780 (+ 2744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:44.667787 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:44.667883 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:44.667998 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:44.670384 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.671521 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.674838 (+ 3317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.675574 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.675691 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:44.689657 (+ 13966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:44.689663 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:44.689666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:44.689703 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:44.693144 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:44.694212 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:44.694218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:44.694222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:44.694416 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:44.695093 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:44.695099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:44.695105 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:44.695419 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:44.695532 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:44.698822 (+ 3290us) server_negotiation.cc:299] Negotiation successful
0927 22:08:44.699094 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:08:44.699368 1991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37207 every 8 connection(s)
I20260927 22:08:44.700188 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:44.663653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33057 (local address 127.0.97.3:38803)
0927 22:08:44.664884 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:44.664921 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:44.667435 (+ 2514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:44.668259 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:44.668275 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:44.669126 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:08:44.670133 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:44.670159 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.671719 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.671725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:44.674631 (+ 2906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:44.674653 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.676086 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.676099 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:44.676501 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:44.685661 (+ 9160us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:08:44.685711 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:08:44.689380 (+ 3669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:08:44.693381 (+ 4001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:44.693394 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:44.693419 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:44.694065 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:44.694555 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:44.694567 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:44.694576 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:44.694943 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:44.695557 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:08:44.695569 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:08:44.696215 (+ 646us) client_negotiation.cc:769] Sending connection context
0927 22:08:44.696633 (+ 418us) client_negotiation.cc:240] Negotiation successful
0927 22:08:44.697386 (+ 753us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"thread_start_us":396,"threads_started":1}
I20260927 22:08:44.700948 1862 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:08:44.703116 1992 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33057
I20260927 22:08:44.704084 1992 heartbeater.cc:464] Registering TS with master...
I20260927 22:08:44.705092 1992 heartbeater.cc:511] Master 127.0.97.62:33057 requested a full tablet report, sending...
I20260927 22:08:44.706769 1512 ts_manager.cc:194] Registered new tserver with Master: e716f1aef97e4c6ea2d5bf5974390f1c (127.0.97.3:37207)
I20260927 22:08:44.708057 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38803
I20260927 22:08:44.708211 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 1862
I20260927 22:08:44.708413 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:08:44.708925 388 external_mini_cluster.cc:1428] Setting key 9a40945e328600866294dc3a8df298a0
I20260927 22:08:44.711913 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:08:44.718510 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:44.706263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59995 (local address 127.0.97.62:33057)
0927 22:08:44.706551 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:44.706555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:44.707062 (+ 507us) server_negotiation.cc:407] Connection header received
0927 22:08:44.707236 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:44.707240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:44.707299 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:44.707375 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:44.708443 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.709462 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.711211 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.711672 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.711790 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:44.713429 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:44.713435 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:44.713440 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:44.713472 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:44.716017 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:44.716649 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:44.716655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:44.716658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:44.716778 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:44.717114 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:44.717121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:44.717124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:44.717405 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:44.717547 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:44.717941 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 22:08:44.718145 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20260927 22:08:44.718672 1998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:44.706273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33057 (local address 127.0.97.3:59995)
0927 22:08:44.706950 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:44.706978 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:44.707113 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:44.707432 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:44.707437 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:44.707962 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:08:44.708289 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:44.708303 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.709597 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.709602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:44.711085 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:44.711096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.711841 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.711848 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:44.712043 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:44.712626 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:08:44.712643 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:08:44.713262 (+ 619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:08:44.716150 (+ 2888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:44.716156 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:44.716161 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:44.716532 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:44.716880 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:44.716887 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:44.716891 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:44.717015 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:44.717513 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:08:44.717518 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:08:44.717689 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 22:08:44.717831 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 22:08:44.718131 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":331,"threads_started":1}
Sep 27 22:08:44 dist-test-slave-mp03 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546924, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:08:44 dist-test-slave-mp03 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546924, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:08:44.759102 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:44.745028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.0.97.62:33057)
0927 22:08:44.745282 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:44.745295 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:44.745467 (+ 172us) server_negotiation.cc:407] Connection header received
0927 22:08:44.745712 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:44.745716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:44.745791 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:44.745899 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:44.747427 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.748157 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.749504 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.749922 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.750057 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:44.753676 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:44.753681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:44.753684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:44.753714 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:44.756499 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:44.757152 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:44.757157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:44.757161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:44.757301 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:44.757662 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:44.757668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:44.757672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:44.757935 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:44.758183 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:44.758472 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 22:08:44.758735 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 22:08:44.763074 1512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:08:44.770406 1512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:08:44.791810 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:44.782171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34847 (local address 127.0.0.1:35910)
0927 22:08:44.783255 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:44.783299 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:44.783508 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:44.785984 (+ 2476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:44.785995 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:44.786043 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:08:44.786477 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:44.786492 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.787898 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.787905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:44.789398 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:44.789409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.790702 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.790709 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:44.790935 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:44.791010 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 22:08:44.791176 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 22:08:44.791391 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":8843,"threads_started":1}
I20260927 22:08:44.792577 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:44.782729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35910 (local address 127.0.97.1:34847)
0927 22:08:44.785492 (+ 2763us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:44.785498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:44.785522 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:08:44.785597 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:44.785602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:44.785785 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:44.785936 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:08:44.786721 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.787765 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.789544 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.790480 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.790635 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:44.790864 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:44.792076 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0927 22:08:44.792274 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2613,"thread_start_us":2485,"threads_started":1}
I20260927 22:08:44.796469 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 398d373d197a452da09af8d16fda618e (DEFAULT_TABLE table=test-table [id=f494f29112ba492896467c91df81b9f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:08:44.798058 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 398d373d197a452da09af8d16fda618e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:44.799135 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:44.783766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37207 (local address 127.0.0.1:33162)
0927 22:08:44.791289 (+ 7523us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:44.791307 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:44.791511 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:44.792447 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:44.792453 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:44.792477 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:08:44.792838 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:44.792853 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.795028 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.795032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:44.796585 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:44.796597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.798207 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.798220 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:44.798435 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:44.798480 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 22:08:44.798568 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 22:08:44.798724 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7400,"thread_start_us":958,"threads_started":1}
I20260927 22:08:44.799230 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:44.782422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41607 (local address 127.0.0.1:57732)
0927 22:08:44.784369 (+ 1947us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:44.784391 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:44.784511 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:44.791703 (+ 7192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:44.791708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:44.791730 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:08:44.792136 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:44.792150 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.794500 (+ 2350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.794508 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:44.796632 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:44.796646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.798469 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.798475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:44.798654 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:44.798690 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:08:44.798757 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 22:08:44.798897 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1781,"thread_start_us":378,"threads_started":1}
I20260927 22:08:44.799944 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:44.784463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33162 (local address 127.0.97.3:37207)
0927 22:08:44.785274 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:44.785287 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:44.791404 (+ 6117us) server_negotiation.cc:407] Connection header received
0927 22:08:44.791706 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:44.791714 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:44.791945 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:44.792121 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:08:44.793051 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.794874 (+ 1823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.796740 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.797776 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.799118 (+ 1342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:44.799384 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:44.799457 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 22:08:44.799635 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":385,"threads_started":1}
I20260927 22:08:44.800107 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:44.782740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57732 (local address 127.0.97.2:41607)
0927 22:08:44.791069 (+ 8329us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:44.791080 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:44.791112 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:08:44.791208 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:44.791215 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:44.791494 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:44.792590 (+ 1096us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:08:44.792820 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.794305 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.796833 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:44.798232 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:44.799359 (+ 1127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:44.799564 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:44.799634 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 22:08:44.799813 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8154,"thread_start_us":8007,"threads_started":1}
I20260927 22:08:44.803363 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 398d373d197a452da09af8d16fda618e (DEFAULT_TABLE table=test-table [id=f494f29112ba492896467c91df81b9f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:08:44.803426 1788 tablet_service.cc:1467] Processing CreateTablet for tablet 398d373d197a452da09af8d16fda618e (DEFAULT_TABLE table=test-table [id=f494f29112ba492896467c91df81b9f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:08:44.804661 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 398d373d197a452da09af8d16fda618e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:44.804708 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 398d373d197a452da09af8d16fda618e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:44.814167 2016 tablet_bootstrap.cc:492] T 398d373d197a452da09af8d16fda618e P a8f11e96ea164f2f8d8792ad228d889b: Bootstrap starting.
I20260927 22:08:44.816202 2017 tablet_bootstrap.cc:492] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c: Bootstrap starting.
I20260927 22:08:44.818054 2016 tablet_bootstrap.cc:654] T 398d373d197a452da09af8d16fda618e P a8f11e96ea164f2f8d8792ad228d889b: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:44.819298 2018 tablet_bootstrap.cc:492] T 398d373d197a452da09af8d16fda618e P 87daa5631c7748caa3118088bbcb7926: Bootstrap starting.
I20260927 22:08:44.819981 2017 tablet_bootstrap.cc:654] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:44.820007 2016 log.cc:824] T 398d373d197a452da09af8d16fda618e P a8f11e96ea164f2f8d8792ad228d889b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:08:44.821180 2017 log.cc:824] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:08:44.822546 2018 tablet_bootstrap.cc:654] T 398d373d197a452da09af8d16fda618e P 87daa5631c7748caa3118088bbcb7926: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:44.823324 2016 tablet_bootstrap.cc:492] T 398d373d197a452da09af8d16fda618e P a8f11e96ea164f2f8d8792ad228d889b: No bootstrap required, opened a new log
I20260927 22:08:44.823614 2016 ts_tablet_manager.cc:1397] T 398d373d197a452da09af8d16fda618e P a8f11e96ea164f2f8d8792ad228d889b: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 22:08:44.823732 2018 log.cc:824] T 398d373d197a452da09af8d16fda618e P 87daa5631c7748caa3118088bbcb7926: Log is configured to *not* fsync() on all Append() calls
I20260927 22:08:44.823983 2017 tablet_bootstrap.cc:492] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c: No bootstrap required, opened a new log
I20260927 22:08:44.824247 2017 ts_tablet_manager.cc:1397] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 22:08:44.827276 2018 tablet_bootstrap.cc:492] T 398d373d197a452da09af8d16fda618e P 87daa5631c7748caa3118088bbcb7926: No bootstrap required, opened a new log
I20260927 22:08:44.827544 2018 ts_tablet_manager.cc:1397] T 398d373d197a452da09af8d16fda618e P 87daa5631c7748caa3118088bbcb7926: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260927 22:08:44.832010 2017 raft_consensus.cc:348] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:44.832501 2017 raft_consensus.cc:374] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:44.832643 2017 raft_consensus.cc:729] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e716f1aef97e4c6ea2d5bf5974390f1c, State: Initialized, Role: FOLLOWER
I20260927 22:08:44.833309 2017 consensus_queue.cc:260] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:44.834517 1992 heartbeater.cc:503] Master 127.0.97.62:33057 was elected leader, sending a full tablet report...
I20260927 22:08:44.834563 2016 raft_consensus.cc:348] T 398d373d197a452da09af8d16fda618e P a8f11e96ea164f2f8d8792ad228d889b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:44.835021 2016 raft_consensus.cc:374] T 398d373d197a452da09af8d16fda618e P a8f11e96ea164f2f8d8792ad228d889b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:44.835115 2016 raft_consensus.cc:729] T 398d373d197a452da09af8d16fda618e P a8f11e96ea164f2f8d8792ad228d889b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8f11e96ea164f2f8d8792ad228d889b, State: Initialized, Role: FOLLOWER
I20260927 22:08:44.835843 2016 consensus_queue.cc:260] T 398d373d197a452da09af8d16fda618e P a8f11e96ea164f2f8d8792ad228d889b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:44.836536 2017 ts_tablet_manager.cc:1428] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260927 22:08:44.838166 2016 ts_tablet_manager.cc:1428] T 398d373d197a452da09af8d16fda618e P a8f11e96ea164f2f8d8792ad228d889b: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260927 22:08:44.840530 2018 raft_consensus.cc:348] T 398d373d197a452da09af8d16fda618e P 87daa5631c7748caa3118088bbcb7926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:44.841048 2018 raft_consensus.cc:374] T 398d373d197a452da09af8d16fda618e P 87daa5631c7748caa3118088bbcb7926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:44.841147 2018 raft_consensus.cc:729] T 398d373d197a452da09af8d16fda618e P 87daa5631c7748caa3118088bbcb7926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87daa5631c7748caa3118088bbcb7926, State: Initialized, Role: FOLLOWER
I20260927 22:08:44.841853 2018 consensus_queue.cc:260] T 398d373d197a452da09af8d16fda618e P 87daa5631c7748caa3118088bbcb7926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:44.843129 1858 heartbeater.cc:503] Master 127.0.97.62:33057 was elected leader, sending a full tablet report...
I20260927 22:08:44.843961 2018 ts_tablet_manager.cc:1428] T 398d373d197a452da09af8d16fda618e P 87daa5631c7748caa3118088bbcb7926: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
W20260927 22:08:44.901656 1720 tablet.cc:2369] T 398d373d197a452da09af8d16fda618e P a8f11e96ea164f2f8d8792ad228d889b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:08:44.937999 1859 tablet.cc:2369] T 398d373d197a452da09af8d16fda618e P 87daa5631c7748caa3118088bbcb7926: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:08:44.953518 1993 tablet.cc:2369] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:08:45.119074 2022 raft_consensus.cc:482] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:08:45.119405 2022 raft_consensus.cc:504] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:45.121165 2022 leader_election.cc:288] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8f11e96ea164f2f8d8792ad228d889b (127.0.97.1:34847), 87daa5631c7748caa3118088bbcb7926 (127.0.97.2:41607)
I20260927 22:08:45.127892 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:45.121438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34847 (local address 127.0.97.3:55263)
0927 22:08:45.121724 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:45.121751 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:45.121976 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:45.122404 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:45.122413 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:45.122435 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:08:45.122722 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:45.122738 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.123979 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.123984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:45.125494 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:45.125506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.127088 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.127096 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:45.127296 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:45.127329 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:08:45.127425 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 22:08:45.127568 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":42}
I20260927 22:08:45.127897 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:45.121565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55263 (local address 127.0.97.1:34847)
0927 22:08:45.121759 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:45.121765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:45.121997 (+ 232us) server_negotiation.cc:407] Connection header received
0927 22:08:45.122155 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:45.122160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:45.122272 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:45.122370 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:08:45.122899 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.123832 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.125654 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.126914 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.127053 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:45.127158 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:45.127482 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 22:08:45.127628 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:08:45.128777 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:45.121651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36037 (local address 127.0.97.2:41607)
0927 22:08:45.121867 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:45.121876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:45.122372 (+ 496us) server_negotiation.cc:407] Connection header received
0927 22:08:45.122561 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:45.122566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:45.122666 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:45.122755 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:08:45.123447 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.124273 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.126252 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.127487 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.127607 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:45.127717 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:45.128389 (+ 672us) server_negotiation.cc:299] Negotiation successful
0927 22:08:45.128527 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:08:45.129004 1674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "398d373d197a452da09af8d16fda618e" candidate_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8f11e96ea164f2f8d8792ad228d889b" is_pre_election: true
I20260927 22:08:45.129504 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:45.121514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41607 (local address 127.0.97.3:36037)
0927 22:08:45.122232 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:45.122252 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:45.122405 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:45.122805 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:45.122813 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:45.122840 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:08:45.123232 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:45.123244 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.124424 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.124428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:45.126075 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:45.126089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.127866 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.127872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:45.128202 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:45.128232 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:08:45.128948 (+ 716us) client_negotiation.cc:240] Negotiation successful
0927 22:08:45.129091 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":4,"thread_start_us":447,"threads_started":1}
I20260927 22:08:45.129624 1674 raft_consensus.cc:2454] T 398d373d197a452da09af8d16fda618e P a8f11e96ea164f2f8d8792ad228d889b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e716f1aef97e4c6ea2d5bf5974390f1c in term 0.
I20260927 22:08:45.130234 1878 leader_election.cc:302] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8f11e96ea164f2f8d8792ad228d889b, e716f1aef97e4c6ea2d5bf5974390f1c; no voters:
I20260927 22:08:45.130656 2022 raft_consensus.cc:2781] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:08:45.130812 2022 raft_consensus.cc:482] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:08:45.130645 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "398d373d197a452da09af8d16fda618e" candidate_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87daa5631c7748caa3118088bbcb7926" is_pre_election: true
I20260927 22:08:45.130918 2022 raft_consensus.cc:3037] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:45.131202 1808 raft_consensus.cc:2454] T 398d373d197a452da09af8d16fda618e P 87daa5631c7748caa3118088bbcb7926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e716f1aef97e4c6ea2d5bf5974390f1c in term 0.
I20260927 22:08:45.133368 2022 raft_consensus.cc:504] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:45.133879 2022 leader_election.cc:288] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c [CANDIDATE]: Term 1 election: Requested vote from peers a8f11e96ea164f2f8d8792ad228d889b (127.0.97.1:34847), 87daa5631c7748caa3118088bbcb7926 (127.0.97.2:41607)
I20260927 22:08:45.134303 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "398d373d197a452da09af8d16fda618e" candidate_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87daa5631c7748caa3118088bbcb7926"
I20260927 22:08:45.134286 1674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "398d373d197a452da09af8d16fda618e" candidate_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8f11e96ea164f2f8d8792ad228d889b"
I20260927 22:08:45.134473 1808 raft_consensus.cc:3037] T 398d373d197a452da09af8d16fda618e P 87daa5631c7748caa3118088bbcb7926 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:45.134495 1674 raft_consensus.cc:3037] T 398d373d197a452da09af8d16fda618e P a8f11e96ea164f2f8d8792ad228d889b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:45.136556 1808 raft_consensus.cc:2454] T 398d373d197a452da09af8d16fda618e P 87daa5631c7748caa3118088bbcb7926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e716f1aef97e4c6ea2d5bf5974390f1c in term 1.
I20260927 22:08:45.136909 1881 leader_election.cc:302] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87daa5631c7748caa3118088bbcb7926, e716f1aef97e4c6ea2d5bf5974390f1c; no voters:
I20260927 22:08:45.137154 1674 raft_consensus.cc:2454] T 398d373d197a452da09af8d16fda618e P a8f11e96ea164f2f8d8792ad228d889b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e716f1aef97e4c6ea2d5bf5974390f1c in term 1.
I20260927 22:08:45.137193 2022 raft_consensus.cc:2781] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:08:45.137689 2022 raft_consensus.cc:686] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c [term 1 LEADER]: Becoming Leader. State: Replica: e716f1aef97e4c6ea2d5bf5974390f1c, State: Running, Role: LEADER
I20260927 22:08:45.138166 2022 consensus_queue.cc:237] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:45.142771 1511 catalog_manager.cc:5495] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c reported cstate change: term changed from 0 to 1, leader changed from <none> to e716f1aef97e4c6ea2d5bf5974390f1c (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } health_report { overall_health: HEALTHY } } }
I20260927 22:08:45.179046 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:45.173925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33166 (local address 127.0.97.3:37207)
0927 22:08:45.174092 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:45.174097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:45.174116 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:08:45.174384 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:45.174387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:45.174460 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:45.174653 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:08:45.174929 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.175638 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.176889 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.177316 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.177403 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:45.177877 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:08:45.178423 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:08:45.178541 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:45.178689 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 22:08:45.178749 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:08:45.187373 1808 raft_consensus.cc:1261] T 398d373d197a452da09af8d16fda618e P 87daa5631c7748caa3118088bbcb7926 [term 1 FOLLOWER]: Refusing update from remote peer e716f1aef97e4c6ea2d5bf5974390f1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:08:45.187368 1674 raft_consensus.cc:1261] T 398d373d197a452da09af8d16fda618e P a8f11e96ea164f2f8d8792ad228d889b [term 1 FOLLOWER]: Refusing update from remote peer e716f1aef97e4c6ea2d5bf5974390f1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:08:45.188373 2022 consensus_queue.cc:1035] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:08:45.188796 2026 consensus_queue.cc:1035] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:08:45.204190 2029 mvcc.cc:205] Tried to move back new op lower bound from 7334080205557157888 to 7334080205369806848. Current Snapshot: MvccSnapshot[applied={T|T < 7334080205557157888}]
I20260927 22:08:45.223353 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:08:45.223691 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:08:45.227341 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b949e81c6e354c868629b02e622250ce: Sending DeleteTablet for 3 replicas of tablet 398d373d197a452da09af8d16fda618e
I20260927 22:08:45.228530 1649 tablet_service.cc:1514] Processing DeleteTablet for tablet 398d373d197a452da09af8d16fda618e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:08:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35910
I20260927 22:08:45.228574 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet 398d373d197a452da09af8d16fda618e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:08:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57732
I20260927 22:08:45.228626 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 398d373d197a452da09af8d16fda618e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:08:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33162
I20260927 22:08:45.231153 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:08:45.231817 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:08:45.240295 2039 tablet_replica.cc:330] T 398d373d197a452da09af8d16fda618e P a8f11e96ea164f2f8d8792ad228d889b: stopping tablet replica
I20260927 22:08:45.240936 2039 raft_consensus.cc:2229] T 398d373d197a452da09af8d16fda618e P a8f11e96ea164f2f8d8792ad228d889b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:08:45.241462 1787 tablet_service.cc:1467] Processing CreateTablet for tablet e93cfc15698647798411182696be8d3c (DEFAULT_TABLE table=test-table [id=12b391271ff8401397c852eb309252d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:08:45.241583 2039 raft_consensus.cc:2258] T 398d373d197a452da09af8d16fda618e P a8f11e96ea164f2f8d8792ad228d889b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:08:45.241592 1926 tablet_service.cc:1467] Processing CreateTablet for tablet e93cfc15698647798411182696be8d3c (DEFAULT_TABLE table=test-table [id=12b391271ff8401397c852eb309252d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:08:45.242038 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93cfc15698647798411182696be8d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:45.242054 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93cfc15698647798411182696be8d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:45.242664 2040 tablet_replica.cc:330] T 398d373d197a452da09af8d16fda618e P 87daa5631c7748caa3118088bbcb7926: stopping tablet replica
I20260927 22:08:45.243117 2040 raft_consensus.cc:2229] T 398d373d197a452da09af8d16fda618e P 87daa5631c7748caa3118088bbcb7926 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:08:45.243604 2040 raft_consensus.cc:2258] T 398d373d197a452da09af8d16fda618e P 87daa5631c7748caa3118088bbcb7926 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:08:45.247822 1649 tablet_service.cc:1467] Processing CreateTablet for tablet e93cfc15698647798411182696be8d3c (DEFAULT_TABLE table=test-table [id=12b391271ff8401397c852eb309252d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:08:45.248247 2017 tablet_bootstrap.cc:492] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c: Bootstrap starting.
I20260927 22:08:45.248286 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93cfc15698647798411182696be8d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:45.250197 2017 tablet_bootstrap.cc:654] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:45.251448 2018 tablet_bootstrap.cc:492] T e93cfc15698647798411182696be8d3c P 87daa5631c7748caa3118088bbcb7926: Bootstrap starting.
I20260927 22:08:45.253152 2018 tablet_bootstrap.cc:654] T e93cfc15698647798411182696be8d3c P 87daa5631c7748caa3118088bbcb7926: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:45.255230 2041 tablet_replica.cc:330] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c: stopping tablet replica
I20260927 22:08:45.255774 2041 raft_consensus.cc:2229] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:08:45.256461 2041 raft_consensus.cc:2258] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:08:45.258484 2041 ts_tablet_manager.cc:1905] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:08:45.260228 2040 ts_tablet_manager.cc:1905] T 398d373d197a452da09af8d16fda618e P 87daa5631c7748caa3118088bbcb7926: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:08:45.260476 2039 ts_tablet_manager.cc:1905] T 398d373d197a452da09af8d16fda618e P a8f11e96ea164f2f8d8792ad228d889b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:08:45.265920 2016 tablet_bootstrap.cc:492] T e93cfc15698647798411182696be8d3c P a8f11e96ea164f2f8d8792ad228d889b: Bootstrap starting.
I20260927 22:08:45.266183 2041 ts_tablet_manager.cc:1918] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:08:45.266333 2041 log.cc:1196] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/398d373d197a452da09af8d16fda618e
I20260927 22:08:45.266880 2041 ts_tablet_manager.cc:1939] T 398d373d197a452da09af8d16fda618e P e716f1aef97e4c6ea2d5bf5974390f1c: Deleting consensus metadata
I20260927 22:08:45.267659 2016 tablet_bootstrap.cc:654] T e93cfc15698647798411182696be8d3c P a8f11e96ea164f2f8d8792ad228d889b: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:45.268575 1499 catalog_manager.cc:4841] TS e716f1aef97e4c6ea2d5bf5974390f1c (127.0.97.3:37207): tablet 398d373d197a452da09af8d16fda618e (table test-table [id=f494f29112ba492896467c91df81b9f6]) successfully deleted
I20260927 22:08:45.274127 2040 ts_tablet_manager.cc:1918] T 398d373d197a452da09af8d16fda618e P 87daa5631c7748caa3118088bbcb7926: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:08:45.274372 2040 log.cc:1196] T 398d373d197a452da09af8d16fda618e P 87daa5631c7748caa3118088bbcb7926: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/398d373d197a452da09af8d16fda618e
I20260927 22:08:45.275040 2040 ts_tablet_manager.cc:1939] T 398d373d197a452da09af8d16fda618e P 87daa5631c7748caa3118088bbcb7926: Deleting consensus metadata
I20260927 22:08:45.277033 1499 catalog_manager.cc:4841] TS 87daa5631c7748caa3118088bbcb7926 (127.0.97.2:41607): tablet 398d373d197a452da09af8d16fda618e (table test-table [id=f494f29112ba492896467c91df81b9f6]) successfully deleted
I20260927 22:08:45.277150 2039 ts_tablet_manager.cc:1918] T 398d373d197a452da09af8d16fda618e P a8f11e96ea164f2f8d8792ad228d889b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:08:45.277434 2039 log.cc:1196] T 398d373d197a452da09af8d16fda618e P a8f11e96ea164f2f8d8792ad228d889b: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-0/wal/wals/398d373d197a452da09af8d16fda618e
I20260927 22:08:45.278129 2039 ts_tablet_manager.cc:1939] T 398d373d197a452da09af8d16fda618e P a8f11e96ea164f2f8d8792ad228d889b: Deleting consensus metadata
I20260927 22:08:45.279199 2016 tablet_bootstrap.cc:492] T e93cfc15698647798411182696be8d3c P a8f11e96ea164f2f8d8792ad228d889b: No bootstrap required, opened a new log
I20260927 22:08:45.279464 2016 ts_tablet_manager.cc:1397] T e93cfc15698647798411182696be8d3c P a8f11e96ea164f2f8d8792ad228d889b: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260927 22:08:45.280284 1496 catalog_manager.cc:4841] TS a8f11e96ea164f2f8d8792ad228d889b (127.0.97.1:34847): tablet 398d373d197a452da09af8d16fda618e (table test-table [id=f494f29112ba492896467c91df81b9f6]) successfully deleted
I20260927 22:08:45.280323 2016 raft_consensus.cc:348] T e93cfc15698647798411182696be8d3c P a8f11e96ea164f2f8d8792ad228d889b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:45.280687 2016 raft_consensus.cc:374] T e93cfc15698647798411182696be8d3c P a8f11e96ea164f2f8d8792ad228d889b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:45.280818 2016 raft_consensus.cc:729] T e93cfc15698647798411182696be8d3c P a8f11e96ea164f2f8d8792ad228d889b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8f11e96ea164f2f8d8792ad228d889b, State: Initialized, Role: FOLLOWER
I20260927 22:08:45.281180 2016 consensus_queue.cc:260] T e93cfc15698647798411182696be8d3c P a8f11e96ea164f2f8d8792ad228d889b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:45.282616 2018 tablet_bootstrap.cc:492] T e93cfc15698647798411182696be8d3c P 87daa5631c7748caa3118088bbcb7926: No bootstrap required, opened a new log
I20260927 22:08:45.282775 2018 ts_tablet_manager.cc:1397] T e93cfc15698647798411182696be8d3c P 87daa5631c7748caa3118088bbcb7926: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.008s
I20260927 22:08:45.282819 2016 ts_tablet_manager.cc:1428] T e93cfc15698647798411182696be8d3c P a8f11e96ea164f2f8d8792ad228d889b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 22:08:45.283720 2018 raft_consensus.cc:348] T e93cfc15698647798411182696be8d3c P 87daa5631c7748caa3118088bbcb7926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:45.283979 2018 raft_consensus.cc:374] T e93cfc15698647798411182696be8d3c P 87daa5631c7748caa3118088bbcb7926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:45.284088 2018 raft_consensus.cc:729] T e93cfc15698647798411182696be8d3c P 87daa5631c7748caa3118088bbcb7926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87daa5631c7748caa3118088bbcb7926, State: Initialized, Role: FOLLOWER
I20260927 22:08:45.284328 2018 consensus_queue.cc:260] T e93cfc15698647798411182696be8d3c P 87daa5631c7748caa3118088bbcb7926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:45.284576 2017 tablet_bootstrap.cc:492] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c: No bootstrap required, opened a new log
I20260927 22:08:45.284742 2017 ts_tablet_manager.cc:1397] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.005s
I20260927 22:08:45.284992 2018 ts_tablet_manager.cc:1428] T e93cfc15698647798411182696be8d3c P 87daa5631c7748caa3118088bbcb7926: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 22:08:45.285692 2017 raft_consensus.cc:348] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:45.285959 2017 raft_consensus.cc:374] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:45.286082 2017 raft_consensus.cc:729] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e716f1aef97e4c6ea2d5bf5974390f1c, State: Initialized, Role: FOLLOWER
I20260927 22:08:45.286414 2017 consensus_queue.cc:260] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:45.287083 2017 ts_tablet_manager.cc:1428] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:08:45.679685 2022 raft_consensus.cc:482] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:08:45.679898 2022 raft_consensus.cc:504] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:45.680526 2022 leader_election.cc:288] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8f11e96ea164f2f8d8792ad228d889b (127.0.97.1:34847), 87daa5631c7748caa3118088bbcb7926 (127.0.97.2:41607)
I20260927 22:08:45.681129 1674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93cfc15698647798411182696be8d3c" candidate_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8f11e96ea164f2f8d8792ad228d889b" is_pre_election: true
I20260927 22:08:45.681147 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93cfc15698647798411182696be8d3c" candidate_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87daa5631c7748caa3118088bbcb7926" is_pre_election: true
I20260927 22:08:45.681355 1674 raft_consensus.cc:2454] T e93cfc15698647798411182696be8d3c P a8f11e96ea164f2f8d8792ad228d889b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e716f1aef97e4c6ea2d5bf5974390f1c in term 0.
I20260927 22:08:45.681383 1808 raft_consensus.cc:2454] T e93cfc15698647798411182696be8d3c P 87daa5631c7748caa3118088bbcb7926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e716f1aef97e4c6ea2d5bf5974390f1c in term 0.
I20260927 22:08:45.681789 1878 leader_election.cc:302] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8f11e96ea164f2f8d8792ad228d889b, e716f1aef97e4c6ea2d5bf5974390f1c; no voters:
I20260927 22:08:45.682029 2022 raft_consensus.cc:2781] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:08:45.682168 2022 raft_consensus.cc:482] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:08:45.682271 2022 raft_consensus.cc:3037] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:45.684391 2022 raft_consensus.cc:504] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:45.684921 2022 leader_election.cc:288] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c [CANDIDATE]: Term 1 election: Requested vote from peers a8f11e96ea164f2f8d8792ad228d889b (127.0.97.1:34847), 87daa5631c7748caa3118088bbcb7926 (127.0.97.2:41607)
I20260927 22:08:45.685338 1674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93cfc15698647798411182696be8d3c" candidate_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8f11e96ea164f2f8d8792ad228d889b"
I20260927 22:08:45.685429 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93cfc15698647798411182696be8d3c" candidate_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87daa5631c7748caa3118088bbcb7926"
I20260927 22:08:45.685530 1674 raft_consensus.cc:3037] T e93cfc15698647798411182696be8d3c P a8f11e96ea164f2f8d8792ad228d889b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:45.685660 1808 raft_consensus.cc:3037] T e93cfc15698647798411182696be8d3c P 87daa5631c7748caa3118088bbcb7926 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:45.687497 1674 raft_consensus.cc:2454] T e93cfc15698647798411182696be8d3c P a8f11e96ea164f2f8d8792ad228d889b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e716f1aef97e4c6ea2d5bf5974390f1c in term 1.
I20260927 22:08:45.687650 1808 raft_consensus.cc:2454] T e93cfc15698647798411182696be8d3c P 87daa5631c7748caa3118088bbcb7926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e716f1aef97e4c6ea2d5bf5974390f1c in term 1.
I20260927 22:08:45.687891 1878 leader_election.cc:302] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8f11e96ea164f2f8d8792ad228d889b, e716f1aef97e4c6ea2d5bf5974390f1c; no voters:
I20260927 22:08:45.688144 2022 raft_consensus.cc:2781] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:08:45.688305 2022 raft_consensus.cc:686] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c [term 1 LEADER]: Becoming Leader. State: Replica: e716f1aef97e4c6ea2d5bf5974390f1c, State: Running, Role: LEADER
I20260927 22:08:45.688565 2022 consensus_queue.cc:237] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:45.690775 1510 catalog_manager.cc:5495] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c reported cstate change: term changed from 0 to 1, leader changed from <none> to e716f1aef97e4c6ea2d5bf5974390f1c (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } health_report { overall_health: HEALTHY } } }
Sep 27 22:08:45 dist-test-slave-mp03 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:08:45.754240 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:45.740859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33057 (local address 127.0.0.1:55680)
0927 22:08:45.741817 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:45.741834 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:45.741950 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:45.742259 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:45.742262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:45.742987 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:08:45.743207 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:45.743216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.744257 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.744260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:45.745203 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:45.745213 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.745834 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.745838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:45.745958 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:45.746579 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:08:45.746644 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:08:45.749315 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:08:45.752092 (+ 2777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:45.752101 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:45.752127 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:45.752599 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:45.752885 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:45.752888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:45.752891 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:45.753019 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:45.753438 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:08:45.753444 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:08:45.753590 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 22:08:45.753723 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 22:08:45.753905 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":640,"threads_started":1}
I20260927 22:08:45.754316 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:45.740859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55680 (local address 127.0.97.62:33057)
0927 22:08:45.741870 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:45.741875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:45.741896 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:08:45.742048 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:45.742052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:45.742118 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:45.742212 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:45.743344 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.744138 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.745314 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.745707 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.745792 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:45.749483 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:45.749488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:45.749491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:45.749518 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:45.751956 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:45.752690 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:45.752694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:45.752696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:45.752803 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:45.753098 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:45.753101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:45.753103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:45.753341 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:45.753442 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:45.753845 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 22:08:45.754015 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":900,"thread_start_us":668,"threads_started":1}
I20260927 22:08:45.759382 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55680:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 22:08:45.760361 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:08:45.766312 1649 tablet_service.cc:1467] Processing CreateTablet for tablet a43b23b56f614902adbd1d411c2ad896 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=09efca90a8c145b4b0a5df85304b29f3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:08:45.766396 1788 tablet_service.cc:1467] Processing CreateTablet for tablet a43b23b56f614902adbd1d411c2ad896 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=09efca90a8c145b4b0a5df85304b29f3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:08:45.766906 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a43b23b56f614902adbd1d411c2ad896. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:45.766906 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a43b23b56f614902adbd1d411c2ad896. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:45.766798 1926 tablet_service.cc:1467] Processing CreateTablet for tablet a43b23b56f614902adbd1d411c2ad896 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=09efca90a8c145b4b0a5df85304b29f3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:08:45.767216 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a43b23b56f614902adbd1d411c2ad896. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:45.772640 2017 tablet_bootstrap.cc:492] T a43b23b56f614902adbd1d411c2ad896 P e716f1aef97e4c6ea2d5bf5974390f1c: Bootstrap starting.
I20260927 22:08:45.774322 2018 tablet_bootstrap.cc:492] T a43b23b56f614902adbd1d411c2ad896 P 87daa5631c7748caa3118088bbcb7926: Bootstrap starting.
I20260927 22:08:45.774765 2017 tablet_bootstrap.cc:654] T a43b23b56f614902adbd1d411c2ad896 P e716f1aef97e4c6ea2d5bf5974390f1c: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:45.775851 2016 tablet_bootstrap.cc:492] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b: Bootstrap starting.
I20260927 22:08:45.775962 2018 tablet_bootstrap.cc:654] T a43b23b56f614902adbd1d411c2ad896 P 87daa5631c7748caa3118088bbcb7926: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:45.777061 2017 tablet_bootstrap.cc:492] T a43b23b56f614902adbd1d411c2ad896 P e716f1aef97e4c6ea2d5bf5974390f1c: No bootstrap required, opened a new log
I20260927 22:08:45.777230 2017 ts_tablet_manager.cc:1397] T a43b23b56f614902adbd1d411c2ad896 P e716f1aef97e4c6ea2d5bf5974390f1c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 22:08:45.777556 2016 tablet_bootstrap.cc:654] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:45.778219 2018 tablet_bootstrap.cc:492] T a43b23b56f614902adbd1d411c2ad896 P 87daa5631c7748caa3118088bbcb7926: No bootstrap required, opened a new log
I20260927 22:08:45.778060 2017 raft_consensus.cc:348] T a43b23b56f614902adbd1d411c2ad896 P e716f1aef97e4c6ea2d5bf5974390f1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:45.778329 2017 raft_consensus.cc:374] T a43b23b56f614902adbd1d411c2ad896 P e716f1aef97e4c6ea2d5bf5974390f1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:45.778369 2018 ts_tablet_manager.cc:1397] T a43b23b56f614902adbd1d411c2ad896 P 87daa5631c7748caa3118088bbcb7926: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260927 22:08:45.778409 2017 raft_consensus.cc:729] T a43b23b56f614902adbd1d411c2ad896 P e716f1aef97e4c6ea2d5bf5974390f1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e716f1aef97e4c6ea2d5bf5974390f1c, State: Initialized, Role: FOLLOWER
I20260927 22:08:45.778669 2017 consensus_queue.cc:260] T a43b23b56f614902adbd1d411c2ad896 P e716f1aef97e4c6ea2d5bf5974390f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:45.779209 2017 ts_tablet_manager.cc:1428] T a43b23b56f614902adbd1d411c2ad896 P e716f1aef97e4c6ea2d5bf5974390f1c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:08:45.779055 2022 tablet_replica.cc:439] T a43b23b56f614902adbd1d411c2ad896 P e716f1aef97e4c6ea2d5bf5974390f1c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } } }
I20260927 22:08:45.779357 2022 tablet_replica.cc:442] T a43b23b56f614902adbd1d411c2ad896 P e716f1aef97e4c6ea2d5bf5974390f1c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:08:45.779436 2018 raft_consensus.cc:348] T a43b23b56f614902adbd1d411c2ad896 P 87daa5631c7748caa3118088bbcb7926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:45.779827 2018 raft_consensus.cc:374] T a43b23b56f614902adbd1d411c2ad896 P 87daa5631c7748caa3118088bbcb7926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:45.779923 2018 raft_consensus.cc:729] T a43b23b56f614902adbd1d411c2ad896 P 87daa5631c7748caa3118088bbcb7926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87daa5631c7748caa3118088bbcb7926, State: Initialized, Role: FOLLOWER
I20260927 22:08:45.780193 2018 consensus_queue.cc:260] T a43b23b56f614902adbd1d411c2ad896 P 87daa5631c7748caa3118088bbcb7926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:45.780624 2024 tablet_replica.cc:439] T a43b23b56f614902adbd1d411c2ad896 P 87daa5631c7748caa3118088bbcb7926: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } } }
I20260927 22:08:45.780886 2024 tablet_replica.cc:442] T a43b23b56f614902adbd1d411c2ad896 P 87daa5631c7748caa3118088bbcb7926: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:08:45.781298 2018 ts_tablet_manager.cc:1428] T a43b23b56f614902adbd1d411c2ad896 P 87daa5631c7748caa3118088bbcb7926: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260927 22:08:45.782980 2016 tablet_bootstrap.cc:492] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b: No bootstrap required, opened a new log
I20260927 22:08:45.783144 2016 ts_tablet_manager.cc:1397] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260927 22:08:45.783864 2016 raft_consensus.cc:348] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:45.784090 2016 raft_consensus.cc:374] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:45.784176 2016 raft_consensus.cc:729] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8f11e96ea164f2f8d8792ad228d889b, State: Initialized, Role: FOLLOWER
I20260927 22:08:45.784408 2016 consensus_queue.cc:260] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:45.785204 2016 ts_tablet_manager.cc:1428] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:08:45.785110 2056 tablet_replica.cc:439] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } } }
I20260927 22:08:45.785419 2056 tablet_replica.cc:442] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:08:45.919065 2056 raft_consensus.cc:482] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:08:45.919409 2056 raft_consensus.cc:504] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:45.921016 2056 leader_election.cc:288] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87daa5631c7748caa3118088bbcb7926 (127.0.97.2:41607), e716f1aef97e4c6ea2d5bf5974390f1c (127.0.97.3:37207)
I20260927 22:08:45.928529 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:45.921481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41607 (local address 127.0.97.1:41033)
0927 22:08:45.922276 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:45.922299 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:45.922465 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:45.923096 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:45.923106 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:45.923134 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:08:45.923539 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:45.923554 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.925002 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.925009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:45.926821 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:45.926839 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.927960 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.927963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:45.928072 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:45.928095 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:08:45.928164 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 22:08:45.928249 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"spinlock_wait_cycles":7552,"thread_start_us":483,"threads_started":1}
I20260927 22:08:45.928625 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:45.921476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41033 (local address 127.0.97.2:41607)
0927 22:08:45.921909 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:45.921913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:45.922384 (+ 471us) server_negotiation.cc:407] Connection header received
0927 22:08:45.922710 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:45.922716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:45.922794 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:45.922909 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:08:45.923740 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.924859 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.926976 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.927825 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.927958 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:45.928060 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:45.928205 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 22:08:45.928332 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":218,"threads_started":1}
I20260927 22:08:45.929103 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43b23b56f614902adbd1d411c2ad896" candidate_uuid: "a8f11e96ea164f2f8d8792ad228d889b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87daa5631c7748caa3118088bbcb7926" is_pre_election: true
I20260927 22:08:45.929352 1808 raft_consensus.cc:2454] T a43b23b56f614902adbd1d411c2ad896 P 87daa5631c7748caa3118088bbcb7926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8f11e96ea164f2f8d8792ad228d889b in term 0.
I20260927 22:08:45.929816 1602 leader_election.cc:302] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87daa5631c7748caa3118088bbcb7926, a8f11e96ea164f2f8d8792ad228d889b; no voters:
I20260927 22:08:45.930043 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:45.922569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60103 (local address 127.0.97.3:37207)
0927 22:08:45.924255 (+ 1686us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:45.924262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:45.924289 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:08:45.924372 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:45.924378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:45.924464 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:45.924569 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:08:45.925128 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.925823 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.927719 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.929076 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.929214 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:45.929311 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:45.929599 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 22:08:45.929741 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1553,"thread_start_us":1424,"threads_started":1}
I20260927 22:08:45.930435 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:45.922476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37207 (local address 127.0.97.1:60103)
0927 22:08:45.923125 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:45.923148 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:45.923327 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:45.924612 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:45.924615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:45.924635 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:08:45.924907 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:45.924913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.925957 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.925962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:45.927571 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:45.927581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.929253 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.929259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:45.929411 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:45.929432 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 22:08:45.929551 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 22:08:45.929676 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":1285,"threads_started":1}
I20260927 22:08:45.930473 2056 raft_consensus.cc:2781] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:08:45.930761 2056 raft_consensus.cc:482] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:08:45.930860 2056 raft_consensus.cc:3037] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:45.931197 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43b23b56f614902adbd1d411c2ad896" candidate_uuid: "a8f11e96ea164f2f8d8792ad228d889b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" is_pre_election: true
I20260927 22:08:45.931573 1947 raft_consensus.cc:2454] T a43b23b56f614902adbd1d411c2ad896 P e716f1aef97e4c6ea2d5bf5974390f1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8f11e96ea164f2f8d8792ad228d889b in term 0.
I20260927 22:08:45.933213 2056 raft_consensus.cc:504] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:45.933826 2056 leader_election.cc:288] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b [CANDIDATE]: Term 1 election: Requested vote from peers 87daa5631c7748caa3118088bbcb7926 (127.0.97.2:41607), e716f1aef97e4c6ea2d5bf5974390f1c (127.0.97.3:37207)
I20260927 22:08:45.934232 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43b23b56f614902adbd1d411c2ad896" candidate_uuid: "a8f11e96ea164f2f8d8792ad228d889b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87daa5631c7748caa3118088bbcb7926"
I20260927 22:08:45.934302 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43b23b56f614902adbd1d411c2ad896" candidate_uuid: "a8f11e96ea164f2f8d8792ad228d889b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c"
I20260927 22:08:45.934434 1808 raft_consensus.cc:3037] T a43b23b56f614902adbd1d411c2ad896 P 87daa5631c7748caa3118088bbcb7926 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:45.934508 1947 raft_consensus.cc:3037] T a43b23b56f614902adbd1d411c2ad896 P e716f1aef97e4c6ea2d5bf5974390f1c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:45.936415 1808 raft_consensus.cc:2454] T a43b23b56f614902adbd1d411c2ad896 P 87daa5631c7748caa3118088bbcb7926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8f11e96ea164f2f8d8792ad228d889b in term 1.
I20260927 22:08:45.936580 1947 raft_consensus.cc:2454] T a43b23b56f614902adbd1d411c2ad896 P e716f1aef97e4c6ea2d5bf5974390f1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8f11e96ea164f2f8d8792ad228d889b in term 1.
I20260927 22:08:45.936836 1602 leader_election.cc:302] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87daa5631c7748caa3118088bbcb7926, a8f11e96ea164f2f8d8792ad228d889b; no voters:
I20260927 22:08:45.937119 2056 raft_consensus.cc:2781] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:08:45.937493 2056 raft_consensus.cc:686] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b [term 1 LEADER]: Becoming Leader. State: Replica: a8f11e96ea164f2f8d8792ad228d889b, State: Running, Role: LEADER
I20260927 22:08:45.937857 2056 consensus_queue.cc:237] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } }
I20260927 22:08:45.939733 2061 tablet_replica.cc:439] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b: TxnStatusTablet state changed. Reason: New leader a8f11e96ea164f2f8d8792ad228d889b. Latest consensus state: current_term: 1 leader_uuid: "a8f11e96ea164f2f8d8792ad228d889b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } } }
I20260927 22:08:45.940140 2061 tablet_replica.cc:442] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:08:45.940896 2063 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 22:08:45.941131 2063 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 22:08:45.941521 1511 catalog_manager.cc:5495] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b reported cstate change: term changed from 0 to 1, leader changed from <none> to a8f11e96ea164f2f8d8792ad228d889b (127.0.97.1). New cstate: current_term: 1 leader_uuid: "a8f11e96ea164f2f8d8792ad228d889b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } health_report { overall_health: UNKNOWN } } }
I20260927 22:08:45.974852 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:45.968612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34847 (local address 127.0.0.1:35912)
0927 22:08:45.968826 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:45.968843 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:45.969002 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:45.969626 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:45.969631 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:45.969642 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:08:45.969887 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:45.969903 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.971430 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.971437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:45.972414 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:45.972425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.973110 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.973116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:45.973271 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:45.973321 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:08:45.974264 (+ 943us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:08:45.974284 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 22:08:45.974367 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 22:08:45.974511 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 22:08:45.974920 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:45.968720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35912 (local address 127.0.97.1:34847)
0927 22:08:45.969293 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:45.969297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:45.969315 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:08:45.969390 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:45.969393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:45.969457 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:45.969614 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:08:45.970054 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.971311 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.972565 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:45.972961 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:45.973086 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:45.973559 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:08:45.974159 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:08:45.974307 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:45.974399 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 22:08:45.974479 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":275,"threads_started":1}
I20260927 22:08:45.979745 1808 raft_consensus.cc:1261] T a43b23b56f614902adbd1d411c2ad896 P 87daa5631c7748caa3118088bbcb7926 [term 1 FOLLOWER]: Refusing update from remote peer a8f11e96ea164f2f8d8792ad228d889b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:08:45.979806 1947 raft_consensus.cc:1261] T a43b23b56f614902adbd1d411c2ad896 P e716f1aef97e4c6ea2d5bf5974390f1c [term 1 FOLLOWER]: Refusing update from remote peer a8f11e96ea164f2f8d8792ad228d889b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:08:45.980487 2061 consensus_queue.cc:1035] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b [LEADER]: Connected to new peer: Peer: permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:08:45.980839 2056 consensus_queue.cc:1035] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:08:45.987433 2024 tablet_replica.cc:439] T a43b23b56f614902adbd1d411c2ad896 P 87daa5631c7748caa3118088bbcb7926: TxnStatusTablet state changed. Reason: New leader a8f11e96ea164f2f8d8792ad228d889b. Latest consensus state: current_term: 1 leader_uuid: "a8f11e96ea164f2f8d8792ad228d889b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } } }
I20260927 22:08:45.987752 2024 tablet_replica.cc:442] T a43b23b56f614902adbd1d411c2ad896 P 87daa5631c7748caa3118088bbcb7926: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:08:45.989784 2067 mvcc.cc:205] Tried to move back new op lower bound from 7334080208805052416 to 7334080208644997120. Current Snapshot: MvccSnapshot[applied={T|T < 7334080208805052416}]
I20260927 22:08:45.991097 2069 mvcc.cc:205] Tried to move back new op lower bound from 7334080208805052416 to 7334080208644997120. Current Snapshot: MvccSnapshot[applied={T|T < 7334080208805052416}]
I20260927 22:08:45.995141 2056 tablet_replica.cc:439] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a8f11e96ea164f2f8d8792ad228d889b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } } }
I20260927 22:08:45.995419 2056 tablet_replica.cc:442] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:08:46.001835 2024 tablet_replica.cc:439] T a43b23b56f614902adbd1d411c2ad896 P 87daa5631c7748caa3118088bbcb7926: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a8f11e96ea164f2f8d8792ad228d889b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } } }
I20260927 22:08:46.002070 2024 tablet_replica.cc:442] T a43b23b56f614902adbd1d411c2ad896 P 87daa5631c7748caa3118088bbcb7926: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:08:46.008057 2022 tablet_replica.cc:439] T a43b23b56f614902adbd1d411c2ad896 P e716f1aef97e4c6ea2d5bf5974390f1c: TxnStatusTablet state changed. Reason: New leader a8f11e96ea164f2f8d8792ad228d889b. Latest consensus state: current_term: 1 leader_uuid: "a8f11e96ea164f2f8d8792ad228d889b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } } }
I20260927 22:08:46.008385 2022 tablet_replica.cc:442] T a43b23b56f614902adbd1d411c2ad896 P e716f1aef97e4c6ea2d5bf5974390f1c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:08:46.010838 2061 tablet_replica.cc:439] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a8f11e96ea164f2f8d8792ad228d889b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } } }
I20260927 22:08:46.011121 2061 tablet_replica.cc:442] T a43b23b56f614902adbd1d411c2ad896 P a8f11e96ea164f2f8d8792ad228d889b: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:08:46.019325 2022 tablet_replica.cc:439] T a43b23b56f614902adbd1d411c2ad896 P e716f1aef97e4c6ea2d5bf5974390f1c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a8f11e96ea164f2f8d8792ad228d889b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 } } peers { permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 } } peers { permanent_uuid: "e716f1aef97e4c6ea2d5bf5974390f1c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37207 } } }
I20260927 22:08:46.019605 2022 tablet_replica.cc:442] T a43b23b56f614902adbd1d411c2ad896 P e716f1aef97e4c6ea2d5bf5974390f1c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:08:46.027709 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:46.020696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37951 (local address 127.0.97.1:34847)
0927 22:08:46.020881 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:46.020888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:46.021508 (+ 620us) server_negotiation.cc:407] Connection header received
0927 22:08:46.021715 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:46.021720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:46.021833 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:46.021915 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:08:46.022456 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:46.023701 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:46.025723 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:46.026369 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:46.026487 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:46.026991 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:08:46.027112 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:08:46.027197 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:46.027340 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 22:08:46.027413 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:08:46.027817 2078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:46.020335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34847 (local address 127.0.97.3:37951)
0927 22:08:46.021415 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:46.021431 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:46.021584 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:46.021940 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:46.021946 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:46.021959 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:08:46.022316 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:46.022328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:46.023830 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:46.023836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:46.025577 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:46.025592 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:46.026543 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:46.026550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:46.026749 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:46.026820 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:08:46.027211 (+ 391us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:08:46.027223 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 22:08:46.027307 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 22:08:46.027444 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"thread_start_us":769,"threads_started":1}
I20260927 22:08:46.037966 1807 raft_consensus.cc:1261] T e93cfc15698647798411182696be8d3c P 87daa5631c7748caa3118088bbcb7926 [term 1 FOLLOWER]: Refusing update from remote peer e716f1aef97e4c6ea2d5bf5974390f1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:08:46.038503 1674 raft_consensus.cc:1261] T e93cfc15698647798411182696be8d3c P a8f11e96ea164f2f8d8792ad228d889b [term 1 FOLLOWER]: Refusing update from remote peer e716f1aef97e4c6ea2d5bf5974390f1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:08:46.039196 2026 consensus_queue.cc:1035] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "87daa5631c7748caa3118088bbcb7926" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:08:46.039448 2022 consensus_queue.cc:1035] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8f11e96ea164f2f8d8792ad228d889b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:08:46.091781 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:46.084572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60395 (local address 127.0.97.3:37207)
0927 22:08:46.084781 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:46.084790 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:46.085594 (+ 804us) server_negotiation.cc:407] Connection header received
0927 22:08:46.085860 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:46.085865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:46.085952 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:46.086060 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:08:46.086711 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:46.087891 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:46.089859 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:46.090493 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:46.090640 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:46.091107 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:08:46.091207 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:08:46.091300 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:46.091406 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 22:08:46.091482 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:08:46.091938 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:46.084580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37207 (local address 127.0.97.1:60395)
0927 22:08:46.085481 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:46.085502 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:46.085677 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:46.086095 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:46.086100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:46.086113 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:08:46.086503 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:46.086516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:46.088023 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:46.088027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:46.089724 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:46.089736 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:46.090690 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:46.090695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:46.090846 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:46.090917 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:08:46.091300 (+ 383us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:08:46.091313 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 22:08:46.091392 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 22:08:46.091524 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":587,"threads_started":1}
I20260927 22:08:46.124229 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:08:46.124495 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:08:46.127225 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b949e81c6e354c868629b02e622250ce: Sending DeleteTablet for 3 replicas of tablet e93cfc15698647798411182696be8d3c
I20260927 22:08:46.127902 1649 tablet_service.cc:1514] Processing DeleteTablet for tablet e93cfc15698647798411182696be8d3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:08:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35910
I20260927 22:08:46.128027 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet e93cfc15698647798411182696be8d3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:08:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33162
I20260927 22:08:46.128074 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet e93cfc15698647798411182696be8d3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:08:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57732
I20260927 22:08:46.139909 2086 tablet_replica.cc:330] T e93cfc15698647798411182696be8d3c P a8f11e96ea164f2f8d8792ad228d889b: stopping tablet replica
I20260927 22:08:46.140201 2086 raft_consensus.cc:2229] T e93cfc15698647798411182696be8d3c P a8f11e96ea164f2f8d8792ad228d889b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:08:46.140409 2087 tablet_replica.cc:330] T e93cfc15698647798411182696be8d3c P 87daa5631c7748caa3118088bbcb7926: stopping tablet replica
I20260927 22:08:46.140512 2086 raft_consensus.cc:2258] T e93cfc15698647798411182696be8d3c P a8f11e96ea164f2f8d8792ad228d889b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:08:46.140616 2087 raft_consensus.cc:2229] T e93cfc15698647798411182696be8d3c P 87daa5631c7748caa3118088bbcb7926 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:08:46.140897 2087 raft_consensus.cc:2258] T e93cfc15698647798411182696be8d3c P 87daa5631c7748caa3118088bbcb7926 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:08:46.141466 2088 tablet_replica.cc:330] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c: stopping tablet replica
I20260927 22:08:46.141660 2088 raft_consensus.cc:2229] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:08:46.141958 2086 ts_tablet_manager.cc:1905] T e93cfc15698647798411182696be8d3c P a8f11e96ea164f2f8d8792ad228d889b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:08:46.142046 2088 raft_consensus.cc:2258] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:08:46.143206 2087 ts_tablet_manager.cc:1905] T e93cfc15698647798411182696be8d3c P 87daa5631c7748caa3118088bbcb7926: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:08:46.150632 2086 ts_tablet_manager.cc:1918] T e93cfc15698647798411182696be8d3c P a8f11e96ea164f2f8d8792ad228d889b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:08:46.150780 2086 log.cc:1196] T e93cfc15698647798411182696be8d3c P a8f11e96ea164f2f8d8792ad228d889b: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-0/wal/wals/e93cfc15698647798411182696be8d3c
I20260927 22:08:46.151283 2086 ts_tablet_manager.cc:1939] T e93cfc15698647798411182696be8d3c P a8f11e96ea164f2f8d8792ad228d889b: Deleting consensus metadata
I20260927 22:08:46.151628 2088 ts_tablet_manager.cc:1905] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:08:46.152472 1496 catalog_manager.cc:4841] TS a8f11e96ea164f2f8d8792ad228d889b (127.0.97.1:34847): tablet e93cfc15698647798411182696be8d3c (table test-table [id=12b391271ff8401397c852eb309252d1]) successfully deleted
I20260927 22:08:46.157054 2088 ts_tablet_manager.cc:1918] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:08:46.157204 2088 log.cc:1196] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/e93cfc15698647798411182696be8d3c
I20260927 22:08:46.157582 2088 ts_tablet_manager.cc:1939] T e93cfc15698647798411182696be8d3c P e716f1aef97e4c6ea2d5bf5974390f1c: Deleting consensus metadata
I20260927 22:08:46.150559 2087 ts_tablet_manager.cc:1918] T e93cfc15698647798411182696be8d3c P 87daa5631c7748caa3118088bbcb7926: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:08:46.158485 2087 log.cc:1196] T e93cfc15698647798411182696be8d3c P 87daa5631c7748caa3118088bbcb7926: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/e93cfc15698647798411182696be8d3c
I20260927 22:08:46.158648 1499 catalog_manager.cc:4841] TS e716f1aef97e4c6ea2d5bf5974390f1c (127.0.97.3:37207): tablet e93cfc15698647798411182696be8d3c (table test-table [id=12b391271ff8401397c852eb309252d1]) successfully deleted
I20260927 22:08:46.158973 2087 ts_tablet_manager.cc:1939] T e93cfc15698647798411182696be8d3c P 87daa5631c7748caa3118088bbcb7926: Deleting consensus metadata
I20260927 22:08:46.159929 1499 catalog_manager.cc:4841] TS 87daa5631c7748caa3118088bbcb7926 (127.0.97.2:41607): tablet e93cfc15698647798411182696be8d3c (table test-table [id=12b391271ff8401397c852eb309252d1]) successfully deleted
Sep 27 22:08:46 dist-test-slave-mp03 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546924, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:08:46.175283 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:46.147197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35916 (local address 127.0.97.1:34847)
0927 22:08:46.147378 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:46.147385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:46.160301 (+ 12916us) server_negotiation.cc:407] Connection header received
0927 22:08:46.160522 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:46.160527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:46.160605 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:46.160724 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:46.162448 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:46.163273 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:46.164555 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:46.164986 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:46.165142 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:46.169287 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:46.169301 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:46.169318 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:46.169355 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:46.172549 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:46.173253 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:46.173260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:46.173263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:46.173381 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:46.173740 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:46.173746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:46.173750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:46.174115 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:46.174427 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:46.174741 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 22:08:46.174968 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 22:08:46.176087 1714 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35916
I20260927 22:08:46.194886 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:46.179347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55686 (local address 127.0.97.62:33057)
0927 22:08:46.179552 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:46.179557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:46.179758 (+ 201us) server_negotiation.cc:407] Connection header received
0927 22:08:46.179913 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:46.179917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:46.179991 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:46.180079 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:46.181690 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:46.182399 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:46.184277 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:46.184952 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:46.185088 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:46.188428 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:46.188434 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:46.188439 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:46.188479 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:46.191999 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:46.192828 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:46.192834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:46.192838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:46.192972 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:46.193367 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:46.193372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:46.193376 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:46.193713 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:46.193940 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:46.194306 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 22:08:46.194542 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260927 22:08:46.195469 1510 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55686
I20260927 22:08:46.197317 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 1583
I20260927 22:08:46.214584 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:46.199497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35930 (local address 127.0.97.1:34847)
0927 22:08:46.199664 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:46.199670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:46.199688 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:08:46.199897 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:46.199904 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:46.199980 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:46.200046 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:46.201677 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:46.202809 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:46.204590 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:46.205302 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:46.205406 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:46.208515 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:46.208522 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:46.208526 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:46.208569 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:46.212091 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:46.212740 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:46.212746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:46.212750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:46.212888 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:46.213205 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:46.213210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:46.213214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:46.213544 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:46.213846 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:46.213962 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 22:08:46.214232 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 22:08:46.215250 1714 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35930
W20260927 22:08:46.215785 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 1583: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:08:46.215967 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 1583
I20260927 22:08:46.231582 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 1723
Sep 27 22:08:46 dist-test-slave-mp03 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546924, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:08:46.244866 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:46.232237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57738 (local address 127.0.97.2:41607)
0927 22:08:46.232403 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:46.232408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:46.232572 (+ 164us) server_negotiation.cc:407] Connection header received
0927 22:08:46.232714 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:46.232718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:46.232779 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:46.232870 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:46.233943 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:46.234686 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:46.235817 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:46.236193 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:46.236276 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:46.239846 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:46.239859 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:46.239874 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:46.239904 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:46.242688 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:46.243233 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:46.243240 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:46.243243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:46.243365 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:46.243643 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:46.243646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:46.243648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:46.243948 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:46.244167 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:46.244338 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 22:08:46.244511 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 22:08:46.245515 1853 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57738
W20260927 22:08:46.246093 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 1723: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:08:46.246232 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 1723
I20260927 22:08:46.262813 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 1862
Sep 27 22:08:46 dist-test-slave-mp03 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546924, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:08:46.277211 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:46.263496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33182 (local address 127.0.97.3:37207)
0927 22:08:46.263700 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:46.263706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:46.263725 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:08:46.263893 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:46.263897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:46.263957 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:46.264063 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:46.265294 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:46.266034 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:46.267275 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:46.267687 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:46.267811 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:46.271874 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:46.271888 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:46.271904 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:46.271935 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:46.274569 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:46.275259 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:46.275266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:46.275268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:46.275390 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:46.275737 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:46.275743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:46.275746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:46.276106 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:46.276419 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:46.276681 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 22:08:46.276892 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 22:08:46.277871 1987 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33182
W20260927 22:08:46.278436 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 1862: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:08:46.278575 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 1862
I20260927 22:08:46.299321 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 1480
I20260927 22:08:46.300379 1552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:08:46.412623 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 1480
2026-09-27T22:08:46Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5073 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:08:46 dist-test-slave-mp03 krb5kdc[2097](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:08:46 dist-test-slave-mp03 krb5kdc[2097](info): set up 2 sockets
Sep 27 22:08:46 dist-test-slave-mp03 krb5kdc[2097](info): commencing operation
krb5kdc: starting...
W20260927 22:08:47.502749 388 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.008s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:08:47 dist-test-slave-mp03 krb5kdc[2097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:08:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:08:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:08:47.686161 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46437
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:08:48.008917 2110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:08:48.009219 2110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:08:48.009310 2110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:08:48.016532 2110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:08:48.016647 2110 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 22:08:48.016688 2110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:08:48.016757 2110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:08:48.016806 2110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:08:48.016860 2110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:08:48.026728 2110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46437
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:08:48.027957 2110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:08:48.029202 2110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:08:48.041028 2116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:48.041018 2115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:48.041260 2118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:08:48.041741 2110 server_base.cc:1034] running on GCE node
I20260927 22:08:48.042663 2110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:08:48.044256 2110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:08:48.045542 2110 hybrid_clock.cc:648] HybridClock initialized: now 1790546928045481 us; error 51 us; skew 500 ppm
I20260927 22:08:48.045924 2110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:08:48 dist-test-slave-mp03 krb5kdc[2097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:08:48.049458 2110 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:08:48.051110 2110 webserver.cc:460] Webserver started at http://127.0.97.62:40709/ using document root <none> and password file <none>
I20260927 22:08:48.051877 2110 fs_manager.cc:359] Metadata directory not provided
I20260927 22:08:48.051967 2110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:08:48.052246 2110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:08:48.055358 2110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "4bc9c4b60ab0473a9677192741a90068"
format_stamp: "Formatted at 2026-09-27 22:08:48 on dist-test-slave-mp03"
server_key: "4da73a1ecca972bb308aba080a60f9b6"
server_key_iv: "b26fcf5cfc6dcf1ba4a75ac4e254bb95"
server_key_version: "encryptionkey@0"
I20260927 22:08:48.056180 2110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "4bc9c4b60ab0473a9677192741a90068"
format_stamp: "Formatted at 2026-09-27 22:08:48 on dist-test-slave-mp03"
server_key: "4da73a1ecca972bb308aba080a60f9b6"
server_key_iv: "b26fcf5cfc6dcf1ba4a75ac4e254bb95"
server_key_version: "encryptionkey@0"
I20260927 22:08:48.061611 2110 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:08:48.065115 2125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:48.066476 2110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:08:48.066682 2110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "4bc9c4b60ab0473a9677192741a90068"
format_stamp: "Formatted at 2026-09-27 22:08:48 on dist-test-slave-mp03"
server_key: "4da73a1ecca972bb308aba080a60f9b6"
server_key_iv: "b26fcf5cfc6dcf1ba4a75ac4e254bb95"
server_key_version: "encryptionkey@0"
I20260927 22:08:48.066905 2110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:08:48.100303 2110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:08:48.104424 2110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:08:48.104700 2110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:08:48.126062 2110 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46437
I20260927 22:08:48.126083 2186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46437 every 8 connection(s)
I20260927 22:08:48.127465 2110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:08:48.131184 2187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:48.132954 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 2110
I20260927 22:08:48.133143 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:08:48.133780 388 external_mini_cluster.cc:1428] Setting key 678d1034e68358911aa09022204ad39c
I20260927 22:08:48.143955 2187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068: Bootstrap starting.
I20260927 22:08:48.147710 2187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:48.149014 2187 log.cc:824] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:08:48 dist-test-slave-mp03 krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:08:48.151870 2187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068: No bootstrap required, opened a new log
I20260927 22:08:48.157974 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:48.135746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.0.97.62:46437)
0927 22:08:48.136565 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:48.136583 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:48.136637 (+ 54us) server_negotiation.cc:407] Connection header received
0927 22:08:48.139326 (+ 2689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:48.139359 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:48.139925 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:48.140339 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:48.141999 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:48.143357 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:48.145067 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:48.145732 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:48.145886 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:48.150742 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:48.150750 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:48.150765 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:48.150798 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:48.153880 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:48.154580 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:48.154586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:48.154635 (+ 49us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:48.154803 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:48.155170 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:48.155176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:48.155183 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:48.155762 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:48.156082 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:48.156492 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 22:08:48.156919 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":382,"threads_started":1}
I20260927 22:08:48.160077 2187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc9c4b60ab0473a9677192741a90068" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46437 } }
I20260927 22:08:48.160450 2187 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:48.160557 2187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc9c4b60ab0473a9677192741a90068, State: Initialized, Role: FOLLOWER
I20260927 22:08:48.161164 2187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc9c4b60ab0473a9677192741a90068" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46437 } }
I20260927 22:08:48.161398 2187 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:08:48.161532 2187 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:08:48.161689 2187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:48.163836 2187 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc9c4b60ab0473a9677192741a90068" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46437 } }
I20260927 22:08:48.164301 2187 leader_election.cc:302] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bc9c4b60ab0473a9677192741a90068; no voters:
I20260927 22:08:48.164826 2187 leader_election.cc:288] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:08:48.164971 2192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:08:48.165679 2192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068 [term 1 LEADER]: Becoming Leader. State: Replica: 4bc9c4b60ab0473a9677192741a90068, State: Running, Role: LEADER
I20260927 22:08:48.166193 2192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc9c4b60ab0473a9677192741a90068" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46437 } }
I20260927 22:08:48.166996 2187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:08:48.169108 2193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bc9c4b60ab0473a9677192741a90068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc9c4b60ab0473a9677192741a90068" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46437 } } }
I20260927 22:08:48.169273 2194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bc9c4b60ab0473a9677192741a90068. Latest consensus state: current_term: 1 leader_uuid: "4bc9c4b60ab0473a9677192741a90068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc9c4b60ab0473a9677192741a90068" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46437 } } }
I20260927 22:08:48.169569 2193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068 [sys.catalog]: This master's current role is: LEADER
I20260927 22:08:48.169622 2194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068 [sys.catalog]: This master's current role is: LEADER
I20260927 22:08:48.172013 2200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:08:48.176920 2200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:08:48.185135 2200 catalog_manager.cc:1347] Generated new cluster ID: 9e8305d7d3bc467cb91e5cbe7facf92c
I20260927 22:08:48.185245 2200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:08:48.209683 2200 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:08:48.211138 2200 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:08:48.219120 2200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068: Generated new TSK 0
I20260927 22:08:48.219933 2200 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:08:48.273912 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46437
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:08:48.613873 2213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:08:48.614202 2213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:08:48.614379 2213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:08:48.621780 2213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:08:48.621934 2213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:08:48.622073 2213 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 22:08:48.622157 2213 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:08:48.622287 2213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:08:48.632289 2213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:08:48.633579 2213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:08:48.634910 2213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:08:48.647784 2218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:48.647809 2219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:48.647833 2221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:08:48.648463 2213 server_base.cc:1034] running on GCE node
I20260927 22:08:48.649026 2213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:08:48.649910 2213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:08:48.651098 2213 hybrid_clock.cc:648] HybridClock initialized: now 1790546928651042 us; error 54 us; skew 500 ppm
I20260927 22:08:48.651409 2213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:08:48 dist-test-slave-mp03 krb5kdc[2097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:08:48.654290 2213 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:08:48.655812 2213 webserver.cc:460] Webserver started at http://127.0.97.1:45879/ using document root <none> and password file <none>
I20260927 22:08:48.656499 2213 fs_manager.cc:359] Metadata directory not provided
I20260927 22:08:48.656631 2213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:08:48.656926 2213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:08:48.659835 2213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "02c9b6626fc84e0281183902da419c29"
format_stamp: "Formatted at 2026-09-27 22:08:48 on dist-test-slave-mp03"
server_key: "bd918f83a7784df6f9ef3b4d21f5f035"
server_key_iv: "ce74fff2fdb213bc488ed69432110299"
server_key_version: "encryptionkey@0"
I20260927 22:08:48.660586 2213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "02c9b6626fc84e0281183902da419c29"
format_stamp: "Formatted at 2026-09-27 22:08:48 on dist-test-slave-mp03"
server_key: "bd918f83a7784df6f9ef3b4d21f5f035"
server_key_iv: "ce74fff2fdb213bc488ed69432110299"
server_key_version: "encryptionkey@0"
I20260927 22:08:48.665477 2213 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 22:08:48.668999 2228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:48.670284 2213 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:08:48.670433 2213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "02c9b6626fc84e0281183902da419c29"
format_stamp: "Formatted at 2026-09-27 22:08:48 on dist-test-slave-mp03"
server_key: "bd918f83a7784df6f9ef3b4d21f5f035"
server_key_iv: "ce74fff2fdb213bc488ed69432110299"
server_key_version: "encryptionkey@0"
I20260927 22:08:48.670663 2213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:08:48.688221 2213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:08:48.692487 2213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:08:48.692766 2213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:08:48.695379 2213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:08:48.695529 2213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:48.695654 2213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:08:48.695745 2213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:08:48 dist-test-slave-mp03 krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:08:48.717801 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:48.695968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50945 (local address 127.0.97.62:46437)
0927 22:08:48.698180 (+ 2212us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:48.698186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:48.698208 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:08:48.699789 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:48.699794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:48.699877 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:48.699997 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:48.702220 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:48.703294 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:48.706314 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:48.706808 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:48.706943 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:48.711121 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:48.711131 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:48.711138 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:48.711186 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:48.714278 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:48.715297 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:48.715304 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:48.715308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:48.715479 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:48.716037 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:48.716043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:48.716046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:48.716325 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:48.716479 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:48.717175 (+ 696us) server_negotiation.cc:299] Negotiation successful
0927 22:08:48.717435 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2084,"thread_start_us":1971,"threads_started":1}
I20260927 22:08:48.719576 2238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:48.696568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46437 (local address 127.0.97.1:50945)
0927 22:08:48.697443 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:48.697495 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:48.699515 (+ 2020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:48.700249 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:48.700261 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:48.700991 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:08:48.701967 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:48.701996 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:48.703457 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:48.703463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:48.706155 (+ 2692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:48.706174 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:48.707139 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:48.707152 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:48.707339 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:48.708133 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:08:48.708154 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:08:48.710868 (+ 2714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:08:48.714449 (+ 3581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:48.714463 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:48.714486 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:48.715140 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:48.715644 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:48.715649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:48.715652 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:48.715908 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:48.716455 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:08:48.716464 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:08:48.716820 (+ 356us) client_negotiation.cc:769] Sending connection context
0927 22:08:48.717093 (+ 273us) client_negotiation.cc:240] Negotiation successful
0927 22:08:48.717615 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":385,"threads_started":1}
I20260927 22:08:48.739584 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:48.725103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37243 (local address 127.0.97.62:46437)
0927 22:08:48.725295 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:48.725304 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:48.726035 (+ 731us) server_negotiation.cc:407] Connection header received
0927 22:08:48.726254 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:48.726262 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:48.726355 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:48.726448 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:48.728041 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:48.729086 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:48.730362 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:48.730857 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:48.730987 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:48.733639 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:48.733649 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:48.733653 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:48.733695 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:48.736724 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:48.737599 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:48.737608 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:48.737612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:48.737744 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:48.738088 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:48.738095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:48.738098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:48.738357 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:48.738509 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:48.739002 (+ 493us) server_negotiation.cc:299] Negotiation successful
0927 22:08:48.739213 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:08:48.739708 2284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:48.725100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46437 (local address 127.0.97.1:37243)
0927 22:08:48.725910 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:48.725937 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:48.726089 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:48.726469 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:48.726477 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:48.727263 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:08:48.727892 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:48.727906 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:48.729205 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:48.729212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:48.730216 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:48.730226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:48.731046 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:48.731053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:48.731239 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:48.731920 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:08:48.731948 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:08:48.733430 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:08:48.736859 (+ 3429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:48.736864 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:48.736871 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:48.737477 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:48.737856 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:48.737861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:48.737865 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:48.737989 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:48.738477 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:08:48.738482 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:08:48.738744 (+ 262us) client_negotiation.cc:769] Sending connection context
0927 22:08:48.738902 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 22:08:48.739264 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":392,"threads_started":1}
I20260927 22:08:48.752542 2213 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33801
I20260927 22:08:48.752564 2348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33801 every 8 connection(s)
I20260927 22:08:48.753724 2213 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:08:48.755718 2349 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46437
I20260927 22:08:48.755971 2349 heartbeater.cc:464] Registering TS with master...
I20260927 22:08:48.756608 2349 heartbeater.cc:511] Master 127.0.97.62:46437 requested a full tablet report, sending...
I20260927 22:08:48.758488 2142 ts_manager.cc:194] Registered new tserver with Master: 02c9b6626fc84e0281183902da419c29 (127.0.97.1:33801)
I20260927 22:08:48.759953 2142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50945
I20260927 22:08:48.763543 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 2213
I20260927 22:08:48.763692 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:08:48.764252 388 external_mini_cluster.cc:1428] Setting key 97bba5a98d5267dcd3c511670bdfda1f
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:08:48.802134 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46437
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:08:49.132853 2353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:08:49.133119 2353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:08:49.133278 2353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:08:49.140936 2353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:08:49.141093 2353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:08:49.141294 2353 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 22:08:49.141395 2353 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:08:49.141541 2353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:08:49.152439 2353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:08:49.153692 2353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:08:49.155066 2353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:08:49.170382 2358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:49.170264 2359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:49.170349 2361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:08:49.171098 2353 server_base.cc:1034] running on GCE node
I20260927 22:08:49.171784 2353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:08:49.172811 2353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:08:49.174041 2353 hybrid_clock.cc:648] HybridClock initialized: now 1790546929173990 us; error 73 us; skew 500 ppm
I20260927 22:08:49.174373 2353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:08:49 dist-test-slave-mp03 krb5kdc[2097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:08:49.185024 2353 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:08:49.186574 2353 webserver.cc:460] Webserver started at http://127.0.97.2:38291/ using document root <none> and password file <none>
I20260927 22:08:49.187419 2353 fs_manager.cc:359] Metadata directory not provided
I20260927 22:08:49.187569 2353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:08:49.187899 2353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:08:49.190951 2353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8"
format_stamp: "Formatted at 2026-09-27 22:08:49 on dist-test-slave-mp03"
server_key: "c58f3b5d09c57a0451362d780a63cc90"
server_key_iv: "040072fd334fa929147c1bcfd4d0cbed"
server_key_version: "encryptionkey@0"
I20260927 22:08:49.191725 2353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8"
format_stamp: "Formatted at 2026-09-27 22:08:49 on dist-test-slave-mp03"
server_key: "c58f3b5d09c57a0451362d780a63cc90"
server_key_iv: "040072fd334fa929147c1bcfd4d0cbed"
server_key_version: "encryptionkey@0"
I20260927 22:08:49.196944 2353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 22:08:49.200618 2368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:49.202219 2353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:08:49.202412 2353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8"
format_stamp: "Formatted at 2026-09-27 22:08:49 on dist-test-slave-mp03"
server_key: "c58f3b5d09c57a0451362d780a63cc90"
server_key_iv: "040072fd334fa929147c1bcfd4d0cbed"
server_key_version: "encryptionkey@0"
I20260927 22:08:49.202642 2353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:08:49.225581 2353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:08:49.230074 2353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:08:49.230456 2353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:08:49.233546 2353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:08:49.233697 2353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:49.233868 2353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:08:49.233973 2353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:08:49 dist-test-slave-mp03 krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:08:49.260032 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:49.233962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55903 (local address 127.0.97.62:46437)
0927 22:08:49.234154 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:49.234160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:49.235802 (+ 1642us) server_negotiation.cc:407] Connection header received
0927 22:08:49.239216 (+ 3414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:49.239224 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:49.239324 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:49.239450 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:49.242301 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.243308 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.245801 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.246714 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.246920 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:49.252322 (+ 5402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:49.252330 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:49.252337 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:49.252380 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:49.255672 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:49.256931 (+ 1259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:49.256935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:49.256937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:49.257055 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:49.257717 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:49.257720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:49.257722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:49.257973 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:49.258087 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:49.259323 (+ 1236us) server_negotiation.cc:299] Negotiation successful
0927 22:08:49.259567 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:08:49.262393 2377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:49.234575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46437 (local address 127.0.97.2:55903)
0927 22:08:49.235619 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:49.235673 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:49.238845 (+ 3172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:49.239792 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:49.239806 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:49.240872 (+ 1066us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:08:49.242068 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:49.242105 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.243451 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.243456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:49.245623 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:49.245644 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.247080 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.247091 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:49.247428 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:49.248143 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:08:49.248174 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:08:49.252006 (+ 3832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:08:49.255824 (+ 3818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:49.255840 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:49.255865 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:49.256768 (+ 903us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:49.257206 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:49.257216 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:49.257223 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:49.257587 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:49.258122 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:08:49.258135 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:08:49.258795 (+ 660us) client_negotiation.cc:769] Sending connection context
0927 22:08:49.259256 (+ 461us) client_negotiation.cc:240] Negotiation successful
0927 22:08:49.260066 (+ 810us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":379,"threads_started":1}
I20260927 22:08:49.282107 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:49.268175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54963 (local address 127.0.97.62:46437)
0927 22:08:49.268345 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:49.268351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:49.269091 (+ 740us) server_negotiation.cc:407] Connection header received
0927 22:08:49.269277 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:49.269281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:49.269337 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:49.269428 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:49.270406 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.271338 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.272633 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.273011 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.273151 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:49.275915 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:49.275927 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:49.275930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:49.275958 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:49.279288 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:49.280125 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:49.280132 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:49.280136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:49.280246 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:49.280585 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:49.280591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:49.280595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:49.280890 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:49.281030 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:49.281500 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 22:08:49.281717 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:08:49.282213 2449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:49.268229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46437 (local address 127.0.97.2:54963)
0927 22:08:49.268998 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:49.269013 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:49.269157 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:49.269453 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:49.269458 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:49.269945 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:08:49.270246 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:49.270254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.271499 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.271505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:49.272491 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:49.272501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.273190 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.273197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:49.273381 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:49.274147 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:08:49.274163 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:08:49.275691 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:08:49.279405 (+ 3714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:49.279411 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:49.279415 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:49.280001 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:49.280346 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:49.280351 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:49.280355 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:49.280479 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:49.281011 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:08:49.281016 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:08:49.281249 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 22:08:49.281417 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 22:08:49.281741 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":471,"threads_started":1}
I20260927 22:08:49.293699 2353 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38195
I20260927 22:08:49.293721 2487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38195 every 8 connection(s)
I20260927 22:08:49.295200 2353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:08:49.297101 2488 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46437
I20260927 22:08:49.297284 2488 heartbeater.cc:464] Registering TS with master...
I20260927 22:08:49.297838 2488 heartbeater.cc:511] Master 127.0.97.62:46437 requested a full tablet report, sending...
I20260927 22:08:49.299059 2142 ts_manager.cc:194] Registered new tserver with Master: 6c310f6d55dc4eabbea8f30ca8ced5e8 (127.0.97.2:38195)
I20260927 22:08:49.299651 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 2353
I20260927 22:08:49.299827 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:08:49.299883 2142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55903
I20260927 22:08:49.300287 388 external_mini_cluster.cc:1428] Setting key efa5117723ef502e7b1c07522049e6ba
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:08:49.341641 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46437
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:08:49.655653 2492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:08:49.656028 2492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:08:49.656219 2492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:08:49.663348 2492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:08:49.663470 2492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:08:49.663604 2492 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 22:08:49.663678 2492 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:08:49.663782 2492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:08:49.673563 2492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:08:49.674949 2492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:08:49.676245 2492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:08:49.689095 2497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:49.688938 2498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:49.689000 2500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:08:49.690225 2492 server_base.cc:1034] running on GCE node
I20260927 22:08:49.690987 2492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:08:49.691983 2492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:08:49.693194 2492 hybrid_clock.cc:648] HybridClock initialized: now 1790546929693126 us; error 64 us; skew 500 ppm
I20260927 22:08:49.693511 2492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:08:49 dist-test-slave-mp03 krb5kdc[2097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:08:49.696614 2492 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:08:49.698062 2492 webserver.cc:460] Webserver started at http://127.0.97.3:46133/ using document root <none> and password file <none>
I20260927 22:08:49.698791 2492 fs_manager.cc:359] Metadata directory not provided
I20260927 22:08:49.698931 2492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:08:49.699245 2492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:08:49.702196 2492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "98cc05014da04d4d9bdd2e29d9d24135"
format_stamp: "Formatted at 2026-09-27 22:08:49 on dist-test-slave-mp03"
server_key: "52d9c16dc491940ff5a20581bf7226a2"
server_key_iv: "aacf903e69c042d734d6b1abe943fc25"
server_key_version: "encryptionkey@0"
I20260927 22:08:49.702991 2492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "98cc05014da04d4d9bdd2e29d9d24135"
format_stamp: "Formatted at 2026-09-27 22:08:49 on dist-test-slave-mp03"
server_key: "52d9c16dc491940ff5a20581bf7226a2"
server_key_iv: "aacf903e69c042d734d6b1abe943fc25"
server_key_version: "encryptionkey@0"
I20260927 22:08:49.708109 2492 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:08:49.711472 2507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:49.712796 2492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:08:49.712951 2492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "98cc05014da04d4d9bdd2e29d9d24135"
format_stamp: "Formatted at 2026-09-27 22:08:49 on dist-test-slave-mp03"
server_key: "52d9c16dc491940ff5a20581bf7226a2"
server_key_iv: "aacf903e69c042d734d6b1abe943fc25"
server_key_version: "encryptionkey@0"
I20260927 22:08:49.713153 2492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:08:49.735567 2492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:08:49.739439 2492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:08:49.739730 2492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:08:49.742144 2492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:08:49.742283 2492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:49.742408 2492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:08:49.742497 2492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:08:49 dist-test-slave-mp03 krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:08:49.762842 2349 heartbeater.cc:503] Master 127.0.97.62:46437 was elected leader, sending a full tablet report...
I20260927 22:08:49.767184 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:49.742949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36361 (local address 127.0.97.62:46437)
0927 22:08:49.743128 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:49.743134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:49.744567 (+ 1433us) server_negotiation.cc:407] Connection header received
0927 22:08:49.747694 (+ 3127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:49.747701 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:49.747803 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:49.747928 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:49.750587 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.751516 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.753953 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.754724 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.754846 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:49.759596 (+ 4750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:49.759605 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:49.759609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:49.759648 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:49.763371 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:49.764440 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:49.764445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:49.764448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:49.764594 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:49.765147 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:49.765152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:49.765156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:49.765508 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:49.765667 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:49.766620 (+ 953us) server_negotiation.cc:299] Negotiation successful
0927 22:08:49.766832 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:08:49.768437 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:49.743498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46437 (local address 127.0.97.3:36361)
0927 22:08:49.744371 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:49.744432 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:49.747317 (+ 2885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:49.748261 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:49.748276 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:49.749245 (+ 969us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:08:49.750344 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:49.750382 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.751674 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.751681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:49.753768 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:49.753782 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.754965 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.754972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:49.755260 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:49.756068 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:08:49.756113 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:08:49.759312 (+ 3199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:08:49.763665 (+ 4353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:49.763674 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:49.763689 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:49.764232 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:49.764732 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:49.764742 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:49.764745 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:49.765003 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:49.765651 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:08:49.765659 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:08:49.766140 (+ 481us) client_negotiation.cc:769] Sending connection context
0927 22:08:49.766526 (+ 386us) client_negotiation.cc:240] Negotiation successful
0927 22:08:49.767100 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":338,"threads_started":1}
I20260927 22:08:49.789278 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:49.775340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52343 (local address 127.0.97.62:46437)
0927 22:08:49.775527 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:49.775534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:49.776484 (+ 950us) server_negotiation.cc:407] Connection header received
0927 22:08:49.776673 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:49.776676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:49.776731 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:49.776805 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:49.778132 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.778871 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.780811 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.781228 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.781352 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:49.783292 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:49.783300 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:49.783307 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:49.783352 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:49.786760 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:49.787441 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:49.787446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:49.787450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:49.787574 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:49.787899 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:49.787905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:49.787908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:49.788189 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:49.788329 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:49.788721 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 22:08:49.788920 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:08:49.789364 2598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:49.775654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46437 (local address 127.0.97.3:52343)
0927 22:08:49.776362 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:49.776392 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:49.776558 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:49.776868 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:49.776875 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:49.777586 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:08:49.777999 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:49.778014 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.779013 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.779018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:49.780673 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:49.780688 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.781401 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.781407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:49.781560 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:49.782162 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:08:49.782178 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:08:49.783112 (+ 934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:08:49.786917 (+ 3805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:49.786925 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:49.786931 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:49.787312 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:49.787679 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:49.787685 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:49.787688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:49.787799 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:49.788296 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:08:49.788301 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:08:49.788467 (+ 166us) client_negotiation.cc:769] Sending connection context
0927 22:08:49.788602 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 22:08:49.788902 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":352,"threads_started":1}
I20260927 22:08:49.794106 2492 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35005
I20260927 22:08:49.794112 2626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35005 every 8 connection(s)
I20260927 22:08:49.795356 2492 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:08:49.797303 2627 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46437
I20260927 22:08:49.797508 2627 heartbeater.cc:464] Registering TS with master...
I20260927 22:08:49.798126 2627 heartbeater.cc:511] Master 127.0.97.62:46437 requested a full tablet report, sending...
I20260927 22:08:49.798702 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 2492
I20260927 22:08:49.798861 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:08:49.799335 388 external_mini_cluster.cc:1428] Setting key 78f3eb47eebbbe25df882fab95580c88
I20260927 22:08:49.799578 2142 ts_manager.cc:194] Registered new tserver with Master: 98cc05014da04d4d9bdd2e29d9d24135 (127.0.97.3:35005)
I20260927 22:08:49.800598 2142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36361
I20260927 22:08:49.802178 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:08:49 dist-test-slave-mp03 krb5kdc[2097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:08:49 dist-test-slave-mp03 krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:08:49.847826 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:49.833204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55912 (local address 127.0.97.62:46437)
0927 22:08:49.833433 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:49.833440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:49.833634 (+ 194us) server_negotiation.cc:407] Connection header received
0927 22:08:49.833798 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:49.833802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:49.833866 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:49.833951 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:49.835554 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.836259 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.837707 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.838091 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.838191 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:49.842707 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:49.842712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:49.842716 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:49.842748 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:49.845265 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:49.845899 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:49.845903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:49.845905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:49.846007 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:49.846356 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:49.846359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:49.846361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:49.846691 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:49.846860 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:49.847281 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 22:08:49.847436 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 22:08:49.851774 2142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:08:49.858323 2142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:08:49.896109 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:49.871258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44416 (local address 127.0.97.1:33801)
0927 22:08:49.887202 (+ 15944us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:49.887215 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:49.887257 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:08:49.887387 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:49.887398 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:49.887769 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:49.888026 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:08:49.889663 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.891357 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.893627 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.894951 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.895108 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:49.895322 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:49.895600 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 22:08:49.895764 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15719,"thread_start_us":15561,"threads_started":1}
I20260927 22:08:49.896139 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:49.870210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33801 (local address 127.0.0.1:44416)
0927 22:08:49.883987 (+ 13777us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:49.884014 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:49.884188 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:49.888824 (+ 4636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:49.888835 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:49.888888 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:08:49.889415 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:49.889432 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.891550 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.891553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:49.893274 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:49.893289 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.895143 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.895149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:49.895356 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:49.895404 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 22:08:49.895545 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 22:08:49.895756 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13581,"thread_start_us":18181,"threads_started":1}
I20260927 22:08:49.898829 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:49.870918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38195 (local address 127.0.0.1:48192)
0927 22:08:49.888729 (+ 17811us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:49.888755 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:49.888859 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:49.891199 (+ 2340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:49.891203 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:49.891226 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:08:49.891614 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:49.891627 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.893396 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.893403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:49.896149 (+ 2746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:49.896163 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.897942 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.897949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:49.898120 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:49.898179 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 22:08:49.898274 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 22:08:49.898404 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17632,"mutex_wait_us":172,"thread_start_us":13351,"threads_started":1}
I20260927 22:08:49.899214 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:49.869799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35005 (local address 127.0.0.1:47216)
0927 22:08:49.877081 (+ 7282us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:49.877120 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:49.877297 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:49.890859 (+ 13562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:49.890864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:49.890892 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:08:49.891290 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:49.891304 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.893370 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.893376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:49.896150 (+ 2774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:49.896163 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.898339 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.898345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:49.898546 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:49.898579 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:08:49.898720 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 22:08:49.898855 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7042,"mutex_wait_us":206,"thread_start_us":6880,"threads_started":1}
I20260927 22:08:49.899597 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:49.871167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48192 (local address 127.0.97.2:38195)
0927 22:08:49.890459 (+ 19292us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:49.890467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:49.890501 (+ 34us) server_negotiation.cc:407] Connection header received
0927 22:08:49.890582 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:49.890589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:49.890842 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:49.891016 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:08:49.891850 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.893255 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.896354 (+ 3099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.897726 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.897856 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:49.898054 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:49.899157 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0927 22:08:49.899318 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19148,"thread_start_us":18991,"threads_started":1}
I20260927 22:08:49.899626 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:49.870154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47216 (local address 127.0.97.3:35005)
0927 22:08:49.890183 (+ 20029us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:49.890194 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:49.890232 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:08:49.890328 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:49.890336 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:49.890646 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:49.890846 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:08:49.891525 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.893213 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.896697 (+ 3484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:49.898093 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:49.898280 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:49.898501 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:49.898761 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 22:08:49.899123 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19851,"thread_start_us":19674,"threads_started":1}
I20260927 22:08:49.900238 2282 tablet_service.cc:1467] Processing CreateTablet for tablet 07f78618d98046439e83889055607f8b (DEFAULT_TABLE table=test-table [id=a042b6c0e7754479a9712c0fd7aefc4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:08:49.901593 2282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07f78618d98046439e83889055607f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:49.902999 2557 tablet_service.cc:1467] Processing CreateTablet for tablet 07f78618d98046439e83889055607f8b (DEFAULT_TABLE table=test-table [id=a042b6c0e7754479a9712c0fd7aefc4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:08:49.902999 2418 tablet_service.cc:1467] Processing CreateTablet for tablet 07f78618d98046439e83889055607f8b (DEFAULT_TABLE table=test-table [id=a042b6c0e7754479a9712c0fd7aefc4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:08:49.904436 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07f78618d98046439e83889055607f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:49.904439 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07f78618d98046439e83889055607f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:49.914475 2646 tablet_bootstrap.cc:492] T 07f78618d98046439e83889055607f8b P 02c9b6626fc84e0281183902da419c29: Bootstrap starting.
I20260927 22:08:49.918138 2646 tablet_bootstrap.cc:654] T 07f78618d98046439e83889055607f8b P 02c9b6626fc84e0281183902da419c29: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:49.919697 2646 log.cc:824] T 07f78618d98046439e83889055607f8b P 02c9b6626fc84e0281183902da419c29: Log is configured to *not* fsync() on all Append() calls
I20260927 22:08:49.923628 2648 tablet_bootstrap.cc:492] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135: Bootstrap starting.
I20260927 22:08:49.924636 2647 tablet_bootstrap.cc:492] T 07f78618d98046439e83889055607f8b P 6c310f6d55dc4eabbea8f30ca8ced5e8: Bootstrap starting.
I20260927 22:08:49.928161 2646 tablet_bootstrap.cc:492] T 07f78618d98046439e83889055607f8b P 02c9b6626fc84e0281183902da419c29: No bootstrap required, opened a new log
I20260927 22:08:49.928432 2646 ts_tablet_manager.cc:1397] T 07f78618d98046439e83889055607f8b P 02c9b6626fc84e0281183902da419c29: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260927 22:08:49.928439 2648 tablet_bootstrap.cc:654] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:49.928450 2647 tablet_bootstrap.cc:654] T 07f78618d98046439e83889055607f8b P 6c310f6d55dc4eabbea8f30ca8ced5e8: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:49.929721 2648 log.cc:824] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135: Log is configured to *not* fsync() on all Append() calls
I20260927 22:08:49.930174 2647 log.cc:824] T 07f78618d98046439e83889055607f8b P 6c310f6d55dc4eabbea8f30ca8ced5e8: Log is configured to *not* fsync() on all Append() calls
I20260927 22:08:49.933368 2648 tablet_bootstrap.cc:492] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135: No bootstrap required, opened a new log
I20260927 22:08:49.933650 2648 ts_tablet_manager.cc:1397] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 22:08:49.933656 2647 tablet_bootstrap.cc:492] T 07f78618d98046439e83889055607f8b P 6c310f6d55dc4eabbea8f30ca8ced5e8: No bootstrap required, opened a new log
I20260927 22:08:49.933946 2647 ts_tablet_manager.cc:1397] T 07f78618d98046439e83889055607f8b P 6c310f6d55dc4eabbea8f30ca8ced5e8: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 22:08:49.938580 2646 raft_consensus.cc:348] T 07f78618d98046439e83889055607f8b P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } }
I20260927 22:08:49.939066 2646 raft_consensus.cc:374] T 07f78618d98046439e83889055607f8b P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:49.939172 2646 raft_consensus.cc:729] T 07f78618d98046439e83889055607f8b P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c9b6626fc84e0281183902da419c29, State: Initialized, Role: FOLLOWER
I20260927 22:08:49.939845 2646 consensus_queue.cc:260] T 07f78618d98046439e83889055607f8b P 02c9b6626fc84e0281183902da419c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } }
I20260927 22:08:49.941635 2646 ts_tablet_manager.cc:1428] T 07f78618d98046439e83889055607f8b P 02c9b6626fc84e0281183902da419c29: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260927 22:08:49.942076 2648 raft_consensus.cc:348] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } }
I20260927 22:08:49.942538 2648 raft_consensus.cc:374] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:49.942700 2648 raft_consensus.cc:729] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98cc05014da04d4d9bdd2e29d9d24135, State: Initialized, Role: FOLLOWER
I20260927 22:08:49.943418 2648 consensus_queue.cc:260] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } }
I20260927 22:08:49.943915 2647 raft_consensus.cc:348] T 07f78618d98046439e83889055607f8b P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } }
I20260927 22:08:49.944346 2647 raft_consensus.cc:374] T 07f78618d98046439e83889055607f8b P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:49.944447 2647 raft_consensus.cc:729] T 07f78618d98046439e83889055607f8b P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c310f6d55dc4eabbea8f30ca8ced5e8, State: Initialized, Role: FOLLOWER
I20260927 22:08:49.945081 2627 heartbeater.cc:503] Master 127.0.97.62:46437 was elected leader, sending a full tablet report...
I20260927 22:08:49.945044 2647 consensus_queue.cc:260] T 07f78618d98046439e83889055607f8b P 6c310f6d55dc4eabbea8f30ca8ced5e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } }
I20260927 22:08:49.945796 2648 ts_tablet_manager.cc:1428] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260927 22:08:49.946671 2488 heartbeater.cc:503] Master 127.0.97.62:46437 was elected leader, sending a full tablet report...
I20260927 22:08:49.947362 2647 ts_tablet_manager.cc:1428] T 07f78618d98046439e83889055607f8b P 6c310f6d55dc4eabbea8f30ca8ced5e8: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
W20260927 22:08:50.006091 2350 tablet.cc:2369] T 07f78618d98046439e83889055607f8b P 02c9b6626fc84e0281183902da419c29: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:08:50.046948 2628 tablet.cc:2369] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:08:50.047314 2489 tablet.cc:2369] T 07f78618d98046439e83889055607f8b P 6c310f6d55dc4eabbea8f30ca8ced5e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:08:50.198127 2653 raft_consensus.cc:482] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:08:50.198477 2653 raft_consensus.cc:504] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } }
I20260927 22:08:50.200285 2653 leader_election.cc:288] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c9b6626fc84e0281183902da419c29 (127.0.97.1:33801), 6c310f6d55dc4eabbea8f30ca8ced5e8 (127.0.97.2:38195)
I20260927 22:08:50.206442 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:50.200534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43235 (local address 127.0.97.1:33801)
0927 22:08:50.200744 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:50.200752 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:50.200943 (+ 191us) server_negotiation.cc:407] Connection header received
0927 22:08:50.201182 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:50.201189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:50.201286 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:50.201383 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:08:50.201825 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.202779 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.204509 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.205523 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.205625 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:50.205707 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:50.206044 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 22:08:50.206156 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 22:08:50.206447 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:50.200469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33801 (local address 127.0.97.3:43235)
0927 22:08:50.200827 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:50.200850 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:50.200995 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:50.201426 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:50.201430 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:50.201450 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:08:50.201682 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:50.201693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.202891 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.202897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:50.204370 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:50.204381 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.205674 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.205680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:50.205876 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:50.205911 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:08:50.206014 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 22:08:50.206147 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":1}
I20260927 22:08:50.207127 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:50.200825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55229 (local address 127.0.97.2:38195)
0927 22:08:50.201012 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:50.201020 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:50.201532 (+ 512us) server_negotiation.cc:407] Connection header received
0927 22:08:50.201676 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:50.201681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:50.201760 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:50.201847 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:08:50.202395 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.203348 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.205175 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.206127 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.206260 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:50.206361 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:50.206745 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 22:08:50.206877 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 22:08:50.207226 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:50.200646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38195 (local address 127.0.97.3:55229)
0927 22:08:50.201426 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:50.201446 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:50.201564 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:50.201875 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:50.201881 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:50.201908 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:08:50.202245 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:50.202255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.203485 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.203489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:50.205017 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:50.205029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.206345 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.206351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:50.206521 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:50.206556 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:08:50.206706 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 22:08:50.206822 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":62,"thread_start_us":429,"threads_started":1}
I20260927 22:08:50.207641 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f78618d98046439e83889055607f8b" candidate_uuid: "98cc05014da04d4d9bdd2e29d9d24135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c9b6626fc84e0281183902da419c29" is_pre_election: true
I20260927 22:08:50.208143 2304 raft_consensus.cc:2454] T 07f78618d98046439e83889055607f8b P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98cc05014da04d4d9bdd2e29d9d24135 in term 0.
I20260927 22:08:50.208077 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f78618d98046439e83889055607f8b" candidate_uuid: "98cc05014da04d4d9bdd2e29d9d24135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" is_pre_election: true
I20260927 22:08:50.208577 2438 raft_consensus.cc:2454] T 07f78618d98046439e83889055607f8b P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98cc05014da04d4d9bdd2e29d9d24135 in term 0.
I20260927 22:08:50.208760 2510 leader_election.cc:302] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c9b6626fc84e0281183902da419c29, 98cc05014da04d4d9bdd2e29d9d24135; no voters:
I20260927 22:08:50.209141 2653 raft_consensus.cc:2781] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:08:50.209287 2653 raft_consensus.cc:482] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:08:50.209384 2653 raft_consensus.cc:3037] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:50.211557 2653 raft_consensus.cc:504] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } }
I20260927 22:08:50.212095 2653 leader_election.cc:288] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135 [CANDIDATE]: Term 1 election: Requested vote from peers 02c9b6626fc84e0281183902da419c29 (127.0.97.1:33801), 6c310f6d55dc4eabbea8f30ca8ced5e8 (127.0.97.2:38195)
I20260927 22:08:50.212486 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f78618d98046439e83889055607f8b" candidate_uuid: "98cc05014da04d4d9bdd2e29d9d24135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c9b6626fc84e0281183902da419c29"
I20260927 22:08:50.212540 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f78618d98046439e83889055607f8b" candidate_uuid: "98cc05014da04d4d9bdd2e29d9d24135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8"
I20260927 22:08:50.212666 2304 raft_consensus.cc:3037] T 07f78618d98046439e83889055607f8b P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:50.212754 2438 raft_consensus.cc:3037] T 07f78618d98046439e83889055607f8b P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:50.214815 2304 raft_consensus.cc:2454] T 07f78618d98046439e83889055607f8b P 02c9b6626fc84e0281183902da419c29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98cc05014da04d4d9bdd2e29d9d24135 in term 1.
I20260927 22:08:50.214923 2438 raft_consensus.cc:2454] T 07f78618d98046439e83889055607f8b P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98cc05014da04d4d9bdd2e29d9d24135 in term 1.
I20260927 22:08:50.215163 2510 leader_election.cc:302] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c9b6626fc84e0281183902da419c29, 98cc05014da04d4d9bdd2e29d9d24135; no voters:
I20260927 22:08:50.215428 2653 raft_consensus.cc:2781] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:08:50.215791 2653 raft_consensus.cc:686] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135 [term 1 LEADER]: Becoming Leader. State: Replica: 98cc05014da04d4d9bdd2e29d9d24135, State: Running, Role: LEADER
I20260927 22:08:50.216286 2653 consensus_queue.cc:237] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } }
I20260927 22:08:50.220554 2141 catalog_manager.cc:5495] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98cc05014da04d4d9bdd2e29d9d24135 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "98cc05014da04d4d9bdd2e29d9d24135" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } health_report { overall_health: UNKNOWN } } }
Sep 27 22:08:50 dist-test-slave-mp03 krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:08:50.285136 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:50.273032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46437 (local address 127.0.0.1:55918)
0927 22:08:50.273628 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:50.273643 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:50.273777 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:50.274050 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:50.274053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:50.274772 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:08:50.275016 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:50.275024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.276020 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.276023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:50.276964 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:50.276974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.277669 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.277673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:50.277781 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:50.278343 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:08:50.278358 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:08:50.280399 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:08:50.282969 (+ 2570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:50.282978 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:50.282984 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:50.283379 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:50.283706 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:50.283709 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:50.283711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:50.283835 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:50.284282 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:08:50.284293 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:08:50.284455 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 22:08:50.284591 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 22:08:50.284792 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":370,"threads_started":1}
I20260927 22:08:50.285202 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:50.272977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55918 (local address 127.0.97.62:46437)
0927 22:08:50.273138 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:50.273157 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:50.273736 (+ 579us) server_negotiation.cc:407] Connection header received
0927 22:08:50.273873 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:50.273877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:50.273935 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:50.274017 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:50.275156 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.275914 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.277076 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.277508 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.277607 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:50.280547 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:50.280551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:50.280554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:50.280580 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:50.282867 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:50.283473 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:50.283477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:50.283479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:50.283611 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:50.283917 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:50.283920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:50.283922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:50.284158 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:50.284258 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:50.284681 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 22:08:50.284920 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":53}
I20260927 22:08:50.289911 2141 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55918:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 22:08:50.290973 2141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:08:50.297642 2282 tablet_service.cc:1467] Processing CreateTablet for tablet 52ac5259e54848618efb4e1e2de7be3b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=261d64b3914f4237a3c90a23fff76376]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:08:50.298005 2418 tablet_service.cc:1467] Processing CreateTablet for tablet 52ac5259e54848618efb4e1e2de7be3b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=261d64b3914f4237a3c90a23fff76376]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:08:50.298581 2282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52ac5259e54848618efb4e1e2de7be3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:50.298064 2557 tablet_service.cc:1467] Processing CreateTablet for tablet 52ac5259e54848618efb4e1e2de7be3b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=261d64b3914f4237a3c90a23fff76376]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:08:50.298900 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52ac5259e54848618efb4e1e2de7be3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:50.299247 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52ac5259e54848618efb4e1e2de7be3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:50.305622 2648 tablet_bootstrap.cc:492] T 52ac5259e54848618efb4e1e2de7be3b P 98cc05014da04d4d9bdd2e29d9d24135: Bootstrap starting.
I20260927 22:08:50.306708 2647 tablet_bootstrap.cc:492] T 52ac5259e54848618efb4e1e2de7be3b P 6c310f6d55dc4eabbea8f30ca8ced5e8: Bootstrap starting.
I20260927 22:08:50.307250 2648 tablet_bootstrap.cc:654] T 52ac5259e54848618efb4e1e2de7be3b P 98cc05014da04d4d9bdd2e29d9d24135: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:50.307374 2646 tablet_bootstrap.cc:492] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29: Bootstrap starting.
I20260927 22:08:50.308530 2647 tablet_bootstrap.cc:654] T 52ac5259e54848618efb4e1e2de7be3b P 6c310f6d55dc4eabbea8f30ca8ced5e8: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:50.308840 2646 tablet_bootstrap.cc:654] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:50.312134 2646 tablet_bootstrap.cc:492] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29: No bootstrap required, opened a new log
I20260927 22:08:50.312284 2646 ts_tablet_manager.cc:1397] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260927 22:08:50.313145 2646 raft_consensus.cc:348] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:50.313416 2646 raft_consensus.cc:374] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:50.313520 2646 raft_consensus.cc:729] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c9b6626fc84e0281183902da419c29, State: Initialized, Role: FOLLOWER
I20260927 22:08:50.313624 2647 tablet_bootstrap.cc:492] T 52ac5259e54848618efb4e1e2de7be3b P 6c310f6d55dc4eabbea8f30ca8ced5e8: No bootstrap required, opened a new log
I20260927 22:08:50.313769 2647 ts_tablet_manager.cc:1397] T 52ac5259e54848618efb4e1e2de7be3b P 6c310f6d55dc4eabbea8f30ca8ced5e8: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260927 22:08:50.313787 2646 consensus_queue.cc:260] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:50.314405 2646 ts_tablet_manager.cc:1428] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260927 22:08:50.314447 2648 tablet_bootstrap.cc:492] T 52ac5259e54848618efb4e1e2de7be3b P 98cc05014da04d4d9bdd2e29d9d24135: No bootstrap required, opened a new log
I20260927 22:08:50.314286 2652 tablet_replica.cc:439] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } }
I20260927 22:08:50.314683 2648 ts_tablet_manager.cc:1397] T 52ac5259e54848618efb4e1e2de7be3b P 98cc05014da04d4d9bdd2e29d9d24135: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260927 22:08:50.314720 2652 tablet_replica.cc:442] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:08:50.315291 2648 raft_consensus.cc:348] T 52ac5259e54848618efb4e1e2de7be3b P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:50.315286 2647 raft_consensus.cc:348] T 52ac5259e54848618efb4e1e2de7be3b P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:50.315486 2648 raft_consensus.cc:374] T 52ac5259e54848618efb4e1e2de7be3b P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:50.315529 2647 raft_consensus.cc:374] T 52ac5259e54848618efb4e1e2de7be3b P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:50.315572 2648 raft_consensus.cc:729] T 52ac5259e54848618efb4e1e2de7be3b P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98cc05014da04d4d9bdd2e29d9d24135, State: Initialized, Role: FOLLOWER
I20260927 22:08:50.315636 2647 raft_consensus.cc:729] T 52ac5259e54848618efb4e1e2de7be3b P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c310f6d55dc4eabbea8f30ca8ced5e8, State: Initialized, Role: FOLLOWER
I20260927 22:08:50.315873 2648 consensus_queue.cc:260] T 52ac5259e54848618efb4e1e2de7be3b P 98cc05014da04d4d9bdd2e29d9d24135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:50.315855 2647 consensus_queue.cc:260] T 52ac5259e54848618efb4e1e2de7be3b P 6c310f6d55dc4eabbea8f30ca8ced5e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:50.316242 2653 tablet_replica.cc:439] T 52ac5259e54848618efb4e1e2de7be3b P 98cc05014da04d4d9bdd2e29d9d24135: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } }
I20260927 22:08:50.316434 2647 ts_tablet_manager.cc:1428] T 52ac5259e54848618efb4e1e2de7be3b P 6c310f6d55dc4eabbea8f30ca8ced5e8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260927 22:08:50.316493 2653 tablet_replica.cc:442] T 52ac5259e54848618efb4e1e2de7be3b P 98cc05014da04d4d9bdd2e29d9d24135: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:08:50.316823 2654 tablet_replica.cc:439] T 52ac5259e54848618efb4e1e2de7be3b P 6c310f6d55dc4eabbea8f30ca8ced5e8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } }
I20260927 22:08:50.317181 2654 tablet_replica.cc:442] T 52ac5259e54848618efb4e1e2de7be3b P 6c310f6d55dc4eabbea8f30ca8ced5e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:08:50.317831 2648 ts_tablet_manager.cc:1428] T 52ac5259e54848618efb4e1e2de7be3b P 98cc05014da04d4d9bdd2e29d9d24135: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260927 22:08:50.437456 2652 raft_consensus.cc:482] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:08:50.437664 2652 raft_consensus.cc:504] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:50.439209 2652 leader_election.cc:288] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c310f6d55dc4eabbea8f30ca8ced5e8 (127.0.97.2:38195), 98cc05014da04d4d9bdd2e29d9d24135 (127.0.97.3:35005)
I20260927 22:08:50.445441 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:50.439492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51861 (local address 127.0.97.2:38195)
0927 22:08:50.439705 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:50.439715 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:50.440245 (+ 530us) server_negotiation.cc:407] Connection header received
0927 22:08:50.440607 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:50.440616 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:50.440710 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:50.440814 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:08:50.441294 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.442003 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.443774 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.444488 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.444636 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:50.444729 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:50.444993 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 22:08:50.445137 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:08:50.445576 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:50.439476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38195 (local address 127.0.97.1:51861)
0927 22:08:50.440127 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:50.440151 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:50.440375 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:50.440834 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:50.440838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:50.440864 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:08:50.441146 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:50.441155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.442124 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.442131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:50.443623 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:50.443634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.444699 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.444705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:50.444844 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:50.444869 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 22:08:50.444969 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 22:08:50.445168 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":56,"thread_start_us":383,"threads_started":1}
I20260927 22:08:50.445765 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:50.439525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35005 (local address 127.0.97.1:53201)
0927 22:08:50.440493 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:50.440512 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:50.440612 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:50.441148 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:50.441152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:50.441180 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:08:50.441470 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:50.441477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.442389 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.442394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:50.443778 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:50.443789 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.444876 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.444882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:50.445082 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:50.445122 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 22:08:50.445219 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 22:08:50.445346 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":1,"thread_start_us":710,"threads_started":1}
I20260927 22:08:50.445876 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:50.439543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53201 (local address 127.0.97.3:35005)
0927 22:08:50.440842 (+ 1299us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:50.440846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:50.440864 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:08:50.440919 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:50.440922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:50.440972 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:50.441114 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:08:50.441609 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.442261 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.443911 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.444673 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.444813 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:50.444917 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:50.445468 (+ 551us) server_negotiation.cc:299] Negotiation successful
0927 22:08:50.445602 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1213,"thread_start_us":1131,"threads_started":1}
I20260927 22:08:50.446583 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52ac5259e54848618efb4e1e2de7be3b" candidate_uuid: "02c9b6626fc84e0281183902da419c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" is_pre_election: true
I20260927 22:08:50.446687 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52ac5259e54848618efb4e1e2de7be3b" candidate_uuid: "02c9b6626fc84e0281183902da419c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98cc05014da04d4d9bdd2e29d9d24135" is_pre_election: true
I20260927 22:08:50.446895 2438 raft_consensus.cc:2454] T 52ac5259e54848618efb4e1e2de7be3b P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c9b6626fc84e0281183902da419c29 in term 0.
I20260927 22:08:50.447064 2577 raft_consensus.cc:2454] T 52ac5259e54848618efb4e1e2de7be3b P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c9b6626fc84e0281183902da419c29 in term 0.
I20260927 22:08:50.447350 2229 leader_election.cc:302] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c9b6626fc84e0281183902da419c29, 6c310f6d55dc4eabbea8f30ca8ced5e8; no voters:
I20260927 22:08:50.447770 2652 raft_consensus.cc:2781] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:08:50.447899 2652 raft_consensus.cc:482] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:08:50.448009 2652 raft_consensus.cc:3037] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:50.450253 2652 raft_consensus.cc:504] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:50.450866 2652 leader_election.cc:288] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29 [CANDIDATE]: Term 1 election: Requested vote from peers 6c310f6d55dc4eabbea8f30ca8ced5e8 (127.0.97.2:38195), 98cc05014da04d4d9bdd2e29d9d24135 (127.0.97.3:35005)
I20260927 22:08:50.451339 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52ac5259e54848618efb4e1e2de7be3b" candidate_uuid: "02c9b6626fc84e0281183902da419c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8"
I20260927 22:08:50.451344 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52ac5259e54848618efb4e1e2de7be3b" candidate_uuid: "02c9b6626fc84e0281183902da419c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98cc05014da04d4d9bdd2e29d9d24135"
I20260927 22:08:50.451541 2438 raft_consensus.cc:3037] T 52ac5259e54848618efb4e1e2de7be3b P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:50.451537 2577 raft_consensus.cc:3037] T 52ac5259e54848618efb4e1e2de7be3b P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:50.453423 2438 raft_consensus.cc:2454] T 52ac5259e54848618efb4e1e2de7be3b P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c9b6626fc84e0281183902da419c29 in term 1.
I20260927 22:08:50.453760 2577 raft_consensus.cc:2454] T 52ac5259e54848618efb4e1e2de7be3b P 98cc05014da04d4d9bdd2e29d9d24135 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c9b6626fc84e0281183902da419c29 in term 1.
I20260927 22:08:50.453816 2229 leader_election.cc:302] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c9b6626fc84e0281183902da419c29, 6c310f6d55dc4eabbea8f30ca8ced5e8; no voters:
I20260927 22:08:50.454087 2652 raft_consensus.cc:2781] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:08:50.454514 2652 raft_consensus.cc:686] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29 [term 1 LEADER]: Becoming Leader. State: Replica: 02c9b6626fc84e0281183902da419c29, State: Running, Role: LEADER
I20260927 22:08:50.455004 2652 consensus_queue.cc:237] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:50.457901 2670 tablet_replica.cc:439] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29: TxnStatusTablet state changed. Reason: New leader 02c9b6626fc84e0281183902da419c29. Latest consensus state: current_term: 1 leader_uuid: "02c9b6626fc84e0281183902da419c29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } }
I20260927 22:08:50.458294 2670 tablet_replica.cc:442] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:08:50.459016 2672 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 22:08:50.459257 2672 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 22:08:50.459810 2140 catalog_manager.cc:5495] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c9b6626fc84e0281183902da419c29 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "02c9b6626fc84e0281183902da419c29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } health_report { overall_health: UNKNOWN } } }
I20260927 22:08:50.510586 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:50.504996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33801 (local address 127.0.0.1:44430)
0927 22:08:50.505257 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:50.505278 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:50.505415 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:50.505940 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:50.505945 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:50.505958 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:08:50.506155 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:50.506166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.507308 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.507316 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:50.508288 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:50.508300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.509003 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.509008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:50.509180 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:50.509231 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:08:50.510062 (+ 831us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:08:50.510081 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 22:08:50.510173 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 22:08:50.510317 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 22:08:50.510658 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:50.505160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44430 (local address 127.0.97.1:33801)
0927 22:08:50.505304 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:50.505309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:50.505325 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:08:50.505688 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:50.505692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:50.505776 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:50.505890 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:08:50.506282 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.507191 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.508429 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.508846 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.508987 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:50.509415 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:08:50.509949 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:08:50.510114 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:50.510209 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 22:08:50.510288 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 22:08:50.518682 2577 raft_consensus.cc:1261] T 52ac5259e54848618efb4e1e2de7be3b P 98cc05014da04d4d9bdd2e29d9d24135 [term 1 FOLLOWER]: Refusing update from remote peer 02c9b6626fc84e0281183902da419c29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:08:50.518935 2438 raft_consensus.cc:1261] T 52ac5259e54848618efb4e1e2de7be3b P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 1 FOLLOWER]: Refusing update from remote peer 02c9b6626fc84e0281183902da419c29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:08:50.519654 2670 consensus_queue.cc:1035] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:08:50.520303 2652 consensus_queue.cc:1035] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:08:50.530356 2653 tablet_replica.cc:439] T 52ac5259e54848618efb4e1e2de7be3b P 98cc05014da04d4d9bdd2e29d9d24135: TxnStatusTablet state changed. Reason: New leader 02c9b6626fc84e0281183902da419c29. Latest consensus state: current_term: 1 leader_uuid: "02c9b6626fc84e0281183902da419c29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } }
I20260927 22:08:50.530714 2653 tablet_replica.cc:442] T 52ac5259e54848618efb4e1e2de7be3b P 98cc05014da04d4d9bdd2e29d9d24135: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:08:50.534032 2652 tablet_replica.cc:439] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02c9b6626fc84e0281183902da419c29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } }
I20260927 22:08:50.534332 2652 tablet_replica.cc:442] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:08:50.534888 2676 mvcc.cc:205] Tried to move back new op lower bound from 7334080227395608576 to 7334080227147255808. Current Snapshot: MvccSnapshot[applied={T|T < 7334080227395608576}]
I20260927 22:08:50.535782 2675 mvcc.cc:205] Tried to move back new op lower bound from 7334080227395608576 to 7334080227147255808. Current Snapshot: MvccSnapshot[applied={T|T < 7334080227395608576 or (T in {7334080227395608576})}]
I20260927 22:08:50.539096 2653 tablet_replica.cc:439] T 52ac5259e54848618efb4e1e2de7be3b P 98cc05014da04d4d9bdd2e29d9d24135: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02c9b6626fc84e0281183902da419c29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } }
I20260927 22:08:50.539541 2653 tablet_replica.cc:442] T 52ac5259e54848618efb4e1e2de7be3b P 98cc05014da04d4d9bdd2e29d9d24135: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:08:50.542977 2654 tablet_replica.cc:439] T 52ac5259e54848618efb4e1e2de7be3b P 6c310f6d55dc4eabbea8f30ca8ced5e8: TxnStatusTablet state changed. Reason: New leader 02c9b6626fc84e0281183902da419c29. Latest consensus state: current_term: 1 leader_uuid: "02c9b6626fc84e0281183902da419c29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } }
I20260927 22:08:50.543275 2654 tablet_replica.cc:442] T 52ac5259e54848618efb4e1e2de7be3b P 6c310f6d55dc4eabbea8f30ca8ced5e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:08:50.544330 2670 tablet_replica.cc:439] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02c9b6626fc84e0281183902da419c29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } }
I20260927 22:08:50.544602 2670 tablet_replica.cc:442] T 52ac5259e54848618efb4e1e2de7be3b P 02c9b6626fc84e0281183902da419c29: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:08:50.558254 2654 tablet_replica.cc:439] T 52ac5259e54848618efb4e1e2de7be3b P 6c310f6d55dc4eabbea8f30ca8ced5e8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02c9b6626fc84e0281183902da419c29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } } }
I20260927 22:08:50.558552 2654 tablet_replica.cc:442] T 52ac5259e54848618efb4e1e2de7be3b P 6c310f6d55dc4eabbea8f30ca8ced5e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:08:50.560621 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:50.544844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47230 (local address 127.0.97.3:35005)
0927 22:08:50.545017 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:50.545024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:50.551273 (+ 6249us) server_negotiation.cc:407] Connection header received
0927 22:08:50.551645 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:50.551654 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:50.551780 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:50.551924 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:08:50.552784 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.554250 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.557030 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.557627 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.557867 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:50.559306 (+ 1439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:08:50.559843 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:08:50.560006 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:50.560243 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 22:08:50.560329 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:08:50.580427 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:50.571873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48329 (local address 127.0.97.1:33801)
0927 22:08:50.572085 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:50.572090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:50.575704 (+ 3614us) server_negotiation.cc:407] Connection header received
0927 22:08:50.575905 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:50.575911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:50.575983 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:50.576091 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:08:50.576551 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.577337 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.578722 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.579153 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.579289 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:50.579688 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:08:50.579804 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:08:50.579896 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:50.580018 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 22:08:50.580102 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 22:08:50.580495 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:50.571779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33801 (local address 127.0.97.3:48329)
0927 22:08:50.575581 (+ 3802us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:50.575602 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:50.575776 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:50.576103 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:50.576108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:50.576118 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:08:50.576414 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:50.576422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.577464 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.577473 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:50.578533 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:50.578553 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.579317 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.579323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:50.579450 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:50.579497 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:08:50.579894 (+ 397us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:08:50.579905 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 22:08:50.579997 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 22:08:50.580132 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3654,"thread_start_us":3539,"threads_started":1}
I20260927 22:08:50.589826 2438 raft_consensus.cc:1261] T 07f78618d98046439e83889055607f8b P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 1 FOLLOWER]: Refusing update from remote peer 98cc05014da04d4d9bdd2e29d9d24135: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:08:50.590161 2304 raft_consensus.cc:1261] T 07f78618d98046439e83889055607f8b P 02c9b6626fc84e0281183902da419c29 [term 1 FOLLOWER]: Refusing update from remote peer 98cc05014da04d4d9bdd2e29d9d24135: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:08:50.590431 2656 consensus_queue.cc:1035] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:08:50.590852 2653 consensus_queue.cc:1035] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:08:50.631165 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:50.624069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55233 (local address 127.0.97.3:35005)
0927 22:08:50.624212 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:50.624216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:50.625136 (+ 920us) server_negotiation.cc:407] Connection header received
0927 22:08:50.625226 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:50.625231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:50.625324 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:50.625415 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:08:50.625952 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.627137 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.629071 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.629769 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.629893 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:50.630374 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:08:50.630512 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:08:50.630626 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:50.630792 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 22:08:50.630865 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 22:08:50.631369 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:50.624129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35005 (local address 127.0.97.1:55233)
0927 22:08:50.624849 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:50.624871 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:50.625028 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:50.625435 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:50.625438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:50.625448 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:08:50.625807 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:50.625817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.627270 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.627275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:50.628942 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:50.628953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:50.629936 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:50.629941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:50.630124 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:50.630176 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:08:50.630628 (+ 452us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:08:50.630642 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 22:08:50.630764 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 22:08:50.630894 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":384,"threads_started":1}
I20260927 22:08:50.672734 2142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:08:50.673038 2142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:08:50.677438 2142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068: Sending DeleteTablet for 3 replicas of tablet 07f78618d98046439e83889055607f8b
I20260927 22:08:50.678716 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet 07f78618d98046439e83889055607f8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:08:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47216
I20260927 22:08:50.679188 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet 07f78618d98046439e83889055607f8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:08:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48192
I20260927 22:08:50.680267 2282 tablet_service.cc:1514] Processing DeleteTablet for tablet 07f78618d98046439e83889055607f8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:08:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44416
I20260927 22:08:50.684314 2142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:08:50.685002 2142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:08:50.692488 2417 tablet_service.cc:1467] Processing CreateTablet for tablet d1f4a5376e134258a33e3d6985297373 (DEFAULT_TABLE table=test-table [id=1ba8f28f67464c06ad227754e1f58084]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:08:50.693116 2417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1f4a5376e134258a33e3d6985297373. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:50.693049 2556 tablet_service.cc:1467] Processing CreateTablet for tablet d1f4a5376e134258a33e3d6985297373 (DEFAULT_TABLE table=test-table [id=1ba8f28f67464c06ad227754e1f58084]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:08:50.693966 2556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1f4a5376e134258a33e3d6985297373. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:50.699144 2278 tablet_service.cc:1467] Processing CreateTablet for tablet d1f4a5376e134258a33e3d6985297373 (DEFAULT_TABLE table=test-table [id=1ba8f28f67464c06ad227754e1f58084]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:08:50.699750 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1f4a5376e134258a33e3d6985297373. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:50.702556 2648 tablet_bootstrap.cc:492] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135: Bootstrap starting.
I20260927 22:08:50.704216 2648 tablet_bootstrap.cc:654] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:50.708930 2698 tablet_replica.cc:330] T 07f78618d98046439e83889055607f8b P 6c310f6d55dc4eabbea8f30ca8ced5e8: stopping tablet replica
I20260927 22:08:50.709673 2646 tablet_bootstrap.cc:492] T d1f4a5376e134258a33e3d6985297373 P 02c9b6626fc84e0281183902da419c29: Bootstrap starting.
I20260927 22:08:50.710753 2698 raft_consensus.cc:2229] T 07f78618d98046439e83889055607f8b P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:08:50.711220 2698 raft_consensus.cc:2258] T 07f78618d98046439e83889055607f8b P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:08:50.711778 2646 tablet_bootstrap.cc:654] T d1f4a5376e134258a33e3d6985297373 P 02c9b6626fc84e0281183902da419c29: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:50.714838 2698 ts_tablet_manager.cc:1905] T 07f78618d98046439e83889055607f8b P 6c310f6d55dc4eabbea8f30ca8ced5e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:08:50.720362 2647 tablet_bootstrap.cc:492] T d1f4a5376e134258a33e3d6985297373 P 6c310f6d55dc4eabbea8f30ca8ced5e8: Bootstrap starting.
I20260927 22:08:50.722733 2647 tablet_bootstrap.cc:654] T d1f4a5376e134258a33e3d6985297373 P 6c310f6d55dc4eabbea8f30ca8ced5e8: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:50.723855 2648 tablet_bootstrap.cc:492] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135: No bootstrap required, opened a new log
I20260927 22:08:50.724056 2648 ts_tablet_manager.cc:1397] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.005s
I20260927 22:08:50.724992 2648 raft_consensus.cc:348] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:50.725294 2648 raft_consensus.cc:374] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:50.725384 2648 raft_consensus.cc:729] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98cc05014da04d4d9bdd2e29d9d24135, State: Initialized, Role: FOLLOWER
I20260927 22:08:50.725661 2648 consensus_queue.cc:260] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:50.727147 2648 ts_tablet_manager.cc:1428] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 22:08:50.736064 2698 ts_tablet_manager.cc:1918] T 07f78618d98046439e83889055607f8b P 6c310f6d55dc4eabbea8f30ca8ced5e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:08:50.741647 2698 log.cc:1196] T 07f78618d98046439e83889055607f8b P 6c310f6d55dc4eabbea8f30ca8ced5e8: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/07f78618d98046439e83889055607f8b
I20260927 22:08:50.742654 2698 ts_tablet_manager.cc:1939] T 07f78618d98046439e83889055607f8b P 6c310f6d55dc4eabbea8f30ca8ced5e8: Deleting consensus metadata
I20260927 22:08:50.743953 2646 tablet_bootstrap.cc:492] T d1f4a5376e134258a33e3d6985297373 P 02c9b6626fc84e0281183902da419c29: No bootstrap required, opened a new log
I20260927 22:08:50.744150 2646 ts_tablet_manager.cc:1397] T d1f4a5376e134258a33e3d6985297373 P 02c9b6626fc84e0281183902da419c29: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.009s
I20260927 22:08:50.744958 2646 raft_consensus.cc:348] T d1f4a5376e134258a33e3d6985297373 P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:50.745229 2646 raft_consensus.cc:374] T d1f4a5376e134258a33e3d6985297373 P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:50.745311 2646 raft_consensus.cc:729] T d1f4a5376e134258a33e3d6985297373 P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c9b6626fc84e0281183902da419c29, State: Initialized, Role: FOLLOWER
I20260927 22:08:50.746488 2647 tablet_bootstrap.cc:492] T d1f4a5376e134258a33e3d6985297373 P 6c310f6d55dc4eabbea8f30ca8ced5e8: No bootstrap required, opened a new log
I20260927 22:08:50.747145 2699 tablet_replica.cc:330] T 07f78618d98046439e83889055607f8b P 02c9b6626fc84e0281183902da419c29: stopping tablet replica
I20260927 22:08:50.747980 2126 catalog_manager.cc:4841] TS 6c310f6d55dc4eabbea8f30ca8ced5e8 (127.0.97.2:38195): tablet 07f78618d98046439e83889055607f8b (table test-table [id=a042b6c0e7754479a9712c0fd7aefc4d]) successfully deleted
I20260927 22:08:50.748325 2647 ts_tablet_manager.cc:1397] T d1f4a5376e134258a33e3d6985297373 P 6c310f6d55dc4eabbea8f30ca8ced5e8: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.010s
I20260927 22:08:50.749112 2647 raft_consensus.cc:348] T d1f4a5376e134258a33e3d6985297373 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:50.749372 2647 raft_consensus.cc:374] T d1f4a5376e134258a33e3d6985297373 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:50.749450 2647 raft_consensus.cc:729] T d1f4a5376e134258a33e3d6985297373 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c310f6d55dc4eabbea8f30ca8ced5e8, State: Initialized, Role: FOLLOWER
I20260927 22:08:50.747628 2699 raft_consensus.cc:2229] T 07f78618d98046439e83889055607f8b P 02c9b6626fc84e0281183902da419c29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:08:50.749989 2699 raft_consensus.cc:2258] T 07f78618d98046439e83889055607f8b P 02c9b6626fc84e0281183902da419c29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:08:50.750757 2646 consensus_queue.cc:260] T d1f4a5376e134258a33e3d6985297373 P 02c9b6626fc84e0281183902da419c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:50.750394 2647 consensus_queue.cc:260] T d1f4a5376e134258a33e3d6985297373 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:50.753126 2646 ts_tablet_manager.cc:1428] T d1f4a5376e134258a33e3d6985297373 P 02c9b6626fc84e0281183902da419c29: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20260927 22:08:50.756098 2647 ts_tablet_manager.cc:1428] T d1f4a5376e134258a33e3d6985297373 P 6c310f6d55dc4eabbea8f30ca8ced5e8: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260927 22:08:50.762738 2699 ts_tablet_manager.cc:1905] T 07f78618d98046439e83889055607f8b P 02c9b6626fc84e0281183902da419c29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:08:50.771340 2699 ts_tablet_manager.cc:1918] T 07f78618d98046439e83889055607f8b P 02c9b6626fc84e0281183902da419c29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:08:50.771632 2699 log.cc:1196] T 07f78618d98046439e83889055607f8b P 02c9b6626fc84e0281183902da419c29: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-0/wal/wals/07f78618d98046439e83889055607f8b
I20260927 22:08:50.772212 2699 ts_tablet_manager.cc:1939] T 07f78618d98046439e83889055607f8b P 02c9b6626fc84e0281183902da419c29: Deleting consensus metadata
I20260927 22:08:50.773068 2697 tablet_replica.cc:330] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135: stopping tablet replica
I20260927 22:08:50.774060 2128 catalog_manager.cc:4841] TS 02c9b6626fc84e0281183902da419c29 (127.0.97.1:33801): tablet 07f78618d98046439e83889055607f8b (table test-table [id=a042b6c0e7754479a9712c0fd7aefc4d]) successfully deleted
I20260927 22:08:50.774701 2697 raft_consensus.cc:2229] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:08:50.779238 2697 raft_consensus.cc:2258] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:08:50.780045 2656 raft_consensus.cc:482] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:08:50.780210 2656 raft_consensus.cc:504] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:50.780864 2656 leader_election.cc:288] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c9b6626fc84e0281183902da419c29 (127.0.97.1:33801), 6c310f6d55dc4eabbea8f30ca8ced5e8 (127.0.97.2:38195)
I20260927 22:08:50.782039 2303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f4a5376e134258a33e3d6985297373" candidate_uuid: "98cc05014da04d4d9bdd2e29d9d24135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c9b6626fc84e0281183902da419c29" is_pre_election: true
I20260927 22:08:50.782349 2303 raft_consensus.cc:2454] T d1f4a5376e134258a33e3d6985297373 P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98cc05014da04d4d9bdd2e29d9d24135 in term 0.
I20260927 22:08:50.782759 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f4a5376e134258a33e3d6985297373" candidate_uuid: "98cc05014da04d4d9bdd2e29d9d24135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" is_pre_election: true
I20260927 22:08:50.783004 2437 raft_consensus.cc:2454] T d1f4a5376e134258a33e3d6985297373 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98cc05014da04d4d9bdd2e29d9d24135 in term 0.
I20260927 22:08:50.783607 2508 leader_election.cc:302] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c310f6d55dc4eabbea8f30ca8ced5e8, 98cc05014da04d4d9bdd2e29d9d24135; no voters:
I20260927 22:08:50.784286 2656 raft_consensus.cc:2781] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:08:50.784381 2656 raft_consensus.cc:482] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:08:50.784490 2656 raft_consensus.cc:3037] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:50.787755 2656 raft_consensus.cc:504] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:50.789050 2303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f4a5376e134258a33e3d6985297373" candidate_uuid: "98cc05014da04d4d9bdd2e29d9d24135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c9b6626fc84e0281183902da419c29"
I20260927 22:08:50.789281 2303 raft_consensus.cc:3037] T d1f4a5376e134258a33e3d6985297373 P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:50.792692 2303 raft_consensus.cc:2454] T d1f4a5376e134258a33e3d6985297373 P 02c9b6626fc84e0281183902da419c29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98cc05014da04d4d9bdd2e29d9d24135 in term 1.
I20260927 22:08:50.798924 2510 leader_election.cc:302] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c9b6626fc84e0281183902da419c29, 98cc05014da04d4d9bdd2e29d9d24135; no voters:
I20260927 22:08:50.799312 2653 raft_consensus.cc:2781] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:08:50.799475 2653 raft_consensus.cc:686] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135 [term 1 LEADER]: Becoming Leader. State: Replica: 98cc05014da04d4d9bdd2e29d9d24135, State: Running, Role: LEADER
I20260927 22:08:50.799767 2653 consensus_queue.cc:237] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:50.802316 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f4a5376e134258a33e3d6985297373" candidate_uuid: "98cc05014da04d4d9bdd2e29d9d24135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8"
I20260927 22:08:50.802529 2437 raft_consensus.cc:3037] T d1f4a5376e134258a33e3d6985297373 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:50.805733 2437 raft_consensus.cc:2454] T d1f4a5376e134258a33e3d6985297373 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98cc05014da04d4d9bdd2e29d9d24135 in term 1.
I20260927 22:08:50.806418 2656 leader_election.cc:288] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135 [CANDIDATE]: Term 1 election: Requested vote from peers 02c9b6626fc84e0281183902da419c29 (127.0.97.1:33801), 6c310f6d55dc4eabbea8f30ca8ced5e8 (127.0.97.2:38195)
I20260927 22:08:50.812774 2697 ts_tablet_manager.cc:1905] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:08:50.819589 2140 catalog_manager.cc:5495] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98cc05014da04d4d9bdd2e29d9d24135 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "98cc05014da04d4d9bdd2e29d9d24135" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } health_report { overall_health: HEALTHY } } }
I20260927 22:08:50.830282 2697 ts_tablet_manager.cc:1918] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:08:50.830497 2697 log.cc:1196] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/07f78618d98046439e83889055607f8b
I20260927 22:08:50.831387 2697 ts_tablet_manager.cc:1939] T 07f78618d98046439e83889055607f8b P 98cc05014da04d4d9bdd2e29d9d24135: Deleting consensus metadata
I20260927 22:08:50.833288 2129 catalog_manager.cc:4841] TS 98cc05014da04d4d9bdd2e29d9d24135 (127.0.97.3:35005): tablet 07f78618d98046439e83889055607f8b (table test-table [id=a042b6c0e7754479a9712c0fd7aefc4d]) successfully deleted
I20260927 22:08:50.870965 2303 raft_consensus.cc:1261] T d1f4a5376e134258a33e3d6985297373 P 02c9b6626fc84e0281183902da419c29 [term 1 FOLLOWER]: Refusing update from remote peer 98cc05014da04d4d9bdd2e29d9d24135: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:08:50.871866 2437 raft_consensus.cc:1261] T d1f4a5376e134258a33e3d6985297373 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 1 FOLLOWER]: Refusing update from remote peer 98cc05014da04d4d9bdd2e29d9d24135: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:08:50.873015 2653 consensus_queue.cc:1035] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260927 22:08:50.873521 2656 consensus_queue.cc:1035] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:08:50.892604 2678 mvcc.cc:205] Tried to move back new op lower bound from 7334080228841291776 to 7334080228558622720. Current Snapshot: MvccSnapshot[applied={T|T < 7334080228841291776}]
W20260927 22:08:50.911903 2676 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 22:08:52.525038 2142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:08:52.525259 2142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:08:52.528976 2142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068: Sending DeleteTablet for 3 replicas of tablet d1f4a5376e134258a33e3d6985297373
I20260927 22:08:52.529775 2282 tablet_service.cc:1514] Processing DeleteTablet for tablet d1f4a5376e134258a33e3d6985297373 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:08:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44416
I20260927 22:08:52.529850 2417 tablet_service.cc:1514] Processing DeleteTablet for tablet d1f4a5376e134258a33e3d6985297373 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:08:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48192
I20260927 22:08:52.529973 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet d1f4a5376e134258a33e3d6985297373 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:08:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47216
I20260927 22:08:52.540604 2730 tablet_replica.cc:330] T d1f4a5376e134258a33e3d6985297373 P 6c310f6d55dc4eabbea8f30ca8ced5e8: stopping tablet replica
I20260927 22:08:52.540742 2729 tablet_replica.cc:330] T d1f4a5376e134258a33e3d6985297373 P 02c9b6626fc84e0281183902da419c29: stopping tablet replica
I20260927 22:08:52.540829 2730 raft_consensus.cc:2229] T d1f4a5376e134258a33e3d6985297373 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:08:52.540920 2729 raft_consensus.cc:2229] T d1f4a5376e134258a33e3d6985297373 P 02c9b6626fc84e0281183902da419c29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:08:52.541085 2730 raft_consensus.cc:2258] T d1f4a5376e134258a33e3d6985297373 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:08:52.541114 2729 raft_consensus.cc:2258] T d1f4a5376e134258a33e3d6985297373 P 02c9b6626fc84e0281183902da419c29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:08:52.542179 2730 ts_tablet_manager.cc:1905] T d1f4a5376e134258a33e3d6985297373 P 6c310f6d55dc4eabbea8f30ca8ced5e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:08:52.542486 2729 ts_tablet_manager.cc:1905] T d1f4a5376e134258a33e3d6985297373 P 02c9b6626fc84e0281183902da419c29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:08:52.545285 2731 tablet_replica.cc:330] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135: stopping tablet replica
I20260927 22:08:52.545534 2731 raft_consensus.cc:2229] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:08:52.545924 2731 raft_consensus.cc:2258] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:08:52.547003 2731 ts_tablet_manager.cc:1905] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:08:52.549072 2729 ts_tablet_manager.cc:1918] T d1f4a5376e134258a33e3d6985297373 P 02c9b6626fc84e0281183902da419c29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:08:52.549072 2730 ts_tablet_manager.cc:1918] T d1f4a5376e134258a33e3d6985297373 P 6c310f6d55dc4eabbea8f30ca8ced5e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:08:52.549204 2729 log.cc:1196] T d1f4a5376e134258a33e3d6985297373 P 02c9b6626fc84e0281183902da419c29: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-0/wal/wals/d1f4a5376e134258a33e3d6985297373
I20260927 22:08:52.549203 2730 log.cc:1196] T d1f4a5376e134258a33e3d6985297373 P 6c310f6d55dc4eabbea8f30ca8ced5e8: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/d1f4a5376e134258a33e3d6985297373
I20260927 22:08:52.549693 2730 ts_tablet_manager.cc:1939] T d1f4a5376e134258a33e3d6985297373 P 6c310f6d55dc4eabbea8f30ca8ced5e8: Deleting consensus metadata
I20260927 22:08:52.549705 2729 ts_tablet_manager.cc:1939] T d1f4a5376e134258a33e3d6985297373 P 02c9b6626fc84e0281183902da419c29: Deleting consensus metadata
I20260927 22:08:52.550788 2126 catalog_manager.cc:4841] TS 6c310f6d55dc4eabbea8f30ca8ced5e8 (127.0.97.2:38195): tablet d1f4a5376e134258a33e3d6985297373 (table test-table [id=1ba8f28f67464c06ad227754e1f58084]) successfully deleted
I20260927 22:08:52.550788 2128 catalog_manager.cc:4841] TS 02c9b6626fc84e0281183902da419c29 (127.0.97.1:33801): tablet d1f4a5376e134258a33e3d6985297373 (table test-table [id=1ba8f28f67464c06ad227754e1f58084]) successfully deleted
I20260927 22:08:52.554037 2731 ts_tablet_manager.cc:1918] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:08:52.554176 2731 log.cc:1196] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/d1f4a5376e134258a33e3d6985297373
I20260927 22:08:52.554548 2731 ts_tablet_manager.cc:1939] T d1f4a5376e134258a33e3d6985297373 P 98cc05014da04d4d9bdd2e29d9d24135: Deleting consensus metadata
I20260927 22:08:52.555402 2129 catalog_manager.cc:4841] TS 98cc05014da04d4d9bdd2e29d9d24135 (127.0.97.3:35005): tablet d1f4a5376e134258a33e3d6985297373 (table test-table [id=1ba8f28f67464c06ad227754e1f58084]) successfully deleted
Sep 27 22:08:52 dist-test-slave-mp03 krb5kdc[2097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546932, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:08:52 dist-test-slave-mp03 krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546932, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:08:52.587167 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:52.573693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55920 (local address 127.0.97.62:46437)
0927 22:08:52.574355 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:52.574361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:52.574387 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:08:52.574631 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:52.574637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:52.574724 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:52.574833 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:52.576157 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:52.576922 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:52.578124 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:52.578528 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:52.578683 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:52.582356 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:52.582361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:52.582364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:52.582394 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:52.584719 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:52.585314 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:52.585321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:52.585324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:52.585442 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:52.585773 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:52.585780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:52.585783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:52.586055 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:52.586283 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:52.586582 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 22:08:52.586812 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":361,"threads_started":1}
I20260927 22:08:52.590246 2142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:08:52.590858 2142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:08:52.596782 2282 tablet_service.cc:1467] Processing CreateTablet for tablet 8d4b56f62a8747619375b1f3a197ce92 (DEFAULT_TABLE table=test-table [id=4f4c1ac5d1264978907ac936c8c3a0b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:08:52.596904 2417 tablet_service.cc:1467] Processing CreateTablet for tablet 8d4b56f62a8747619375b1f3a197ce92 (DEFAULT_TABLE table=test-table [id=4f4c1ac5d1264978907ac936c8c3a0b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:08:52.596932 2557 tablet_service.cc:1467] Processing CreateTablet for tablet 8d4b56f62a8747619375b1f3a197ce92 (DEFAULT_TABLE table=test-table [id=4f4c1ac5d1264978907ac936c8c3a0b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:08:52.597229 2282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d4b56f62a8747619375b1f3a197ce92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:52.597304 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d4b56f62a8747619375b1f3a197ce92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:52.597324 2417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d4b56f62a8747619375b1f3a197ce92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:52.603358 2740 tablet_bootstrap.cc:492] T 8d4b56f62a8747619375b1f3a197ce92 P 02c9b6626fc84e0281183902da419c29: Bootstrap starting.
I20260927 22:08:52.604635 2740 tablet_bootstrap.cc:654] T 8d4b56f62a8747619375b1f3a197ce92 P 02c9b6626fc84e0281183902da419c29: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:52.612207 2742 tablet_bootstrap.cc:492] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135: Bootstrap starting.
I20260927 22:08:52.612401 2741 tablet_bootstrap.cc:492] T 8d4b56f62a8747619375b1f3a197ce92 P 6c310f6d55dc4eabbea8f30ca8ced5e8: Bootstrap starting.
I20260927 22:08:52.612439 2740 tablet_bootstrap.cc:492] T 8d4b56f62a8747619375b1f3a197ce92 P 02c9b6626fc84e0281183902da419c29: No bootstrap required, opened a new log
I20260927 22:08:52.612582 2740 ts_tablet_manager.cc:1397] T 8d4b56f62a8747619375b1f3a197ce92 P 02c9b6626fc84e0281183902da419c29: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260927 22:08:52.613869 2741 tablet_bootstrap.cc:654] T 8d4b56f62a8747619375b1f3a197ce92 P 6c310f6d55dc4eabbea8f30ca8ced5e8: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:52.613780 2740 raft_consensus.cc:348] T 8d4b56f62a8747619375b1f3a197ce92 P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:52.614040 2740 raft_consensus.cc:374] T 8d4b56f62a8747619375b1f3a197ce92 P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:52.614140 2740 raft_consensus.cc:729] T 8d4b56f62a8747619375b1f3a197ce92 P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c9b6626fc84e0281183902da419c29, State: Initialized, Role: FOLLOWER
I20260927 22:08:52.614435 2742 tablet_bootstrap.cc:654] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:52.614405 2740 consensus_queue.cc:260] T 8d4b56f62a8747619375b1f3a197ce92 P 02c9b6626fc84e0281183902da419c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:52.615023 2740 ts_tablet_manager.cc:1428] T 8d4b56f62a8747619375b1f3a197ce92 P 02c9b6626fc84e0281183902da419c29: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:08:52.617376 2741 tablet_bootstrap.cc:492] T 8d4b56f62a8747619375b1f3a197ce92 P 6c310f6d55dc4eabbea8f30ca8ced5e8: No bootstrap required, opened a new log
I20260927 22:08:52.617491 2742 tablet_bootstrap.cc:492] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135: No bootstrap required, opened a new log
I20260927 22:08:52.617540 2741 ts_tablet_manager.cc:1397] T 8d4b56f62a8747619375b1f3a197ce92 P 6c310f6d55dc4eabbea8f30ca8ced5e8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 22:08:52.617617 2742 ts_tablet_manager.cc:1397] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 22:08:52.618302 2741 raft_consensus.cc:348] T 8d4b56f62a8747619375b1f3a197ce92 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:52.618342 2742 raft_consensus.cc:348] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:52.618541 2741 raft_consensus.cc:374] T 8d4b56f62a8747619375b1f3a197ce92 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:52.618561 2742 raft_consensus.cc:374] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:52.618669 2742 raft_consensus.cc:729] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98cc05014da04d4d9bdd2e29d9d24135, State: Initialized, Role: FOLLOWER
I20260927 22:08:52.618661 2741 raft_consensus.cc:729] T 8d4b56f62a8747619375b1f3a197ce92 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c310f6d55dc4eabbea8f30ca8ced5e8, State: Initialized, Role: FOLLOWER
I20260927 22:08:52.618911 2741 consensus_queue.cc:260] T 8d4b56f62a8747619375b1f3a197ce92 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:52.618896 2742 consensus_queue.cc:260] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:52.619494 2742 ts_tablet_manager.cc:1428] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:08:52.619728 2741 ts_tablet_manager.cc:1428] T 8d4b56f62a8747619375b1f3a197ce92 P 6c310f6d55dc4eabbea8f30ca8ced5e8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 22:08:52.677258 2718 raft_consensus.cc:482] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:08:52.677491 2718 raft_consensus.cc:504] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:52.678069 2718 leader_election.cc:288] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c9b6626fc84e0281183902da419c29 (127.0.97.1:33801), 6c310f6d55dc4eabbea8f30ca8ced5e8 (127.0.97.2:38195)
I20260927 22:08:52.678568 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4b56f62a8747619375b1f3a197ce92" candidate_uuid: "98cc05014da04d4d9bdd2e29d9d24135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" is_pre_election: true
I20260927 22:08:52.678654 2303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4b56f62a8747619375b1f3a197ce92" candidate_uuid: "98cc05014da04d4d9bdd2e29d9d24135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c9b6626fc84e0281183902da419c29" is_pre_election: true
I20260927 22:08:52.678853 2437 raft_consensus.cc:2454] T 8d4b56f62a8747619375b1f3a197ce92 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98cc05014da04d4d9bdd2e29d9d24135 in term 0.
I20260927 22:08:52.678900 2303 raft_consensus.cc:2454] T 8d4b56f62a8747619375b1f3a197ce92 P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98cc05014da04d4d9bdd2e29d9d24135 in term 0.
I20260927 22:08:52.679312 2508 leader_election.cc:302] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c310f6d55dc4eabbea8f30ca8ced5e8, 98cc05014da04d4d9bdd2e29d9d24135; no voters:
I20260927 22:08:52.679569 2718 raft_consensus.cc:2781] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:08:52.679665 2718 raft_consensus.cc:482] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:08:52.679728 2718 raft_consensus.cc:3037] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:52.681680 2718 raft_consensus.cc:504] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:52.682222 2718 leader_election.cc:288] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135 [CANDIDATE]: Term 1 election: Requested vote from peers 02c9b6626fc84e0281183902da419c29 (127.0.97.1:33801), 6c310f6d55dc4eabbea8f30ca8ced5e8 (127.0.97.2:38195)
I20260927 22:08:52.682677 2303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4b56f62a8747619375b1f3a197ce92" candidate_uuid: "98cc05014da04d4d9bdd2e29d9d24135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c9b6626fc84e0281183902da419c29"
I20260927 22:08:52.682881 2303 raft_consensus.cc:3037] T 8d4b56f62a8747619375b1f3a197ce92 P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:52.682901 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4b56f62a8747619375b1f3a197ce92" candidate_uuid: "98cc05014da04d4d9bdd2e29d9d24135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8"
I20260927 22:08:52.683140 2437 raft_consensus.cc:3037] T 8d4b56f62a8747619375b1f3a197ce92 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:52.684839 2303 raft_consensus.cc:2454] T 8d4b56f62a8747619375b1f3a197ce92 P 02c9b6626fc84e0281183902da419c29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98cc05014da04d4d9bdd2e29d9d24135 in term 1.
I20260927 22:08:52.685089 2437 raft_consensus.cc:2454] T 8d4b56f62a8747619375b1f3a197ce92 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98cc05014da04d4d9bdd2e29d9d24135 in term 1.
I20260927 22:08:52.685252 2510 leader_election.cc:302] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c9b6626fc84e0281183902da419c29, 98cc05014da04d4d9bdd2e29d9d24135; no voters:
I20260927 22:08:52.685524 2718 raft_consensus.cc:2781] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:08:52.685640 2718 raft_consensus.cc:686] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135 [term 1 LEADER]: Becoming Leader. State: Replica: 98cc05014da04d4d9bdd2e29d9d24135, State: Running, Role: LEADER
I20260927 22:08:52.685880 2718 consensus_queue.cc:237] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:52.688369 2142 catalog_manager.cc:5495] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98cc05014da04d4d9bdd2e29d9d24135 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "98cc05014da04d4d9bdd2e29d9d24135" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } health_report { overall_health: HEALTHY } } }
I20260927 22:08:52.712410 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:52.707158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47236 (local address 127.0.97.3:35005)
0927 22:08:52.707713 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:52.707717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:52.707733 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:08:52.707777 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:52.707780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:52.707834 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:52.707932 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:08:52.708705 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:52.709436 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:52.710740 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:52.711174 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:52.711303 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:52.711714 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:08:52.711817 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:08:52.711908 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:52.712051 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 22:08:52.712120 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":303,"threads_started":1}
I20260927 22:08:52.719980 2437 raft_consensus.cc:1261] T 8d4b56f62a8747619375b1f3a197ce92 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 1 FOLLOWER]: Refusing update from remote peer 98cc05014da04d4d9bdd2e29d9d24135: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:08:52.719949 2303 raft_consensus.cc:1261] T 8d4b56f62a8747619375b1f3a197ce92 P 02c9b6626fc84e0281183902da419c29 [term 1 FOLLOWER]: Refusing update from remote peer 98cc05014da04d4d9bdd2e29d9d24135: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:08:52.720467 2718 consensus_queue.cc:1035] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:08:52.720750 2722 consensus_queue.cc:1035] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:08:52.765245 2140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:08:52.765439 2140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:08:52.768256 2140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068: Sending DeleteTablet for 3 replicas of tablet 8d4b56f62a8747619375b1f3a197ce92
I20260927 22:08:52.769029 2417 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d4b56f62a8747619375b1f3a197ce92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:08:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48192
I20260927 22:08:52.769382 2730 tablet_replica.cc:330] T 8d4b56f62a8747619375b1f3a197ce92 P 6c310f6d55dc4eabbea8f30ca8ced5e8: stopping tablet replica
I20260927 22:08:52.769443 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d4b56f62a8747619375b1f3a197ce92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:08:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47216
I20260927 22:08:52.769591 2730 raft_consensus.cc:2229] T 8d4b56f62a8747619375b1f3a197ce92 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:08:52.769685 2731 tablet_replica.cc:330] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135: stopping tablet replica
I20260927 22:08:52.769786 2730 raft_consensus.cc:2258] T 8d4b56f62a8747619375b1f3a197ce92 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:08:52.769896 2731 raft_consensus.cc:2229] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:08:52.769913 2282 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d4b56f62a8747619375b1f3a197ce92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:08:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44416
I20260927 22:08:52.770190 2731 raft_consensus.cc:2258] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:08:52.770493 2729 tablet_replica.cc:330] T 8d4b56f62a8747619375b1f3a197ce92 P 02c9b6626fc84e0281183902da419c29: stopping tablet replica
I20260927 22:08:52.770716 2729 raft_consensus.cc:2229] T 8d4b56f62a8747619375b1f3a197ce92 P 02c9b6626fc84e0281183902da419c29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:08:52.770910 2729 raft_consensus.cc:2258] T 8d4b56f62a8747619375b1f3a197ce92 P 02c9b6626fc84e0281183902da419c29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:08:52.770788 2140 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 22:08:52.771183 2730 ts_tablet_manager.cc:1905] T 8d4b56f62a8747619375b1f3a197ce92 P 6c310f6d55dc4eabbea8f30ca8ced5e8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 22:08:52.771582 2140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:08:52.772154 2729 ts_tablet_manager.cc:1905] T 8d4b56f62a8747619375b1f3a197ce92 P 02c9b6626fc84e0281183902da419c29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:08:52.772174 2731 ts_tablet_manager.cc:1905] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:08:52.781219 2282 tablet_service.cc:1467] Processing CreateTablet for tablet 52bf5fd2b97449dbb00092b1e5a7f399 (DEFAULT_TABLE table=test-table [id=f2f66c81a4e7424a948f56340e3bb234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:08:52.781441 2557 tablet_service.cc:1467] Processing CreateTablet for tablet 52bf5fd2b97449dbb00092b1e5a7f399 (DEFAULT_TABLE table=test-table [id=f2f66c81a4e7424a948f56340e3bb234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:08:52.781647 2417 tablet_service.cc:1467] Processing CreateTablet for tablet 52bf5fd2b97449dbb00092b1e5a7f399 (DEFAULT_TABLE table=test-table [id=f2f66c81a4e7424a948f56340e3bb234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:08:52.782130 2417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52bf5fd2b97449dbb00092b1e5a7f399. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:52.805975 2282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52bf5fd2b97449dbb00092b1e5a7f399. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:52.806048 2731 ts_tablet_manager.cc:1918] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:08:52.806241 2731 log.cc:1196] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/8d4b56f62a8747619375b1f3a197ce92
I20260927 22:08:52.806759 2731 ts_tablet_manager.cc:1939] T 8d4b56f62a8747619375b1f3a197ce92 P 98cc05014da04d4d9bdd2e29d9d24135: Deleting consensus metadata
I20260927 22:08:52.807942 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52bf5fd2b97449dbb00092b1e5a7f399. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:08:52.810228 2729 ts_tablet_manager.cc:1918] T 8d4b56f62a8747619375b1f3a197ce92 P 02c9b6626fc84e0281183902da419c29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:08:52.810366 2729 log.cc:1196] T 8d4b56f62a8747619375b1f3a197ce92 P 02c9b6626fc84e0281183902da419c29: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-0/wal/wals/8d4b56f62a8747619375b1f3a197ce92
I20260927 22:08:52.812855 2729 ts_tablet_manager.cc:1939] T 8d4b56f62a8747619375b1f3a197ce92 P 02c9b6626fc84e0281183902da419c29: Deleting consensus metadata
I20260927 22:08:52.813041 2129 catalog_manager.cc:4841] TS 98cc05014da04d4d9bdd2e29d9d24135 (127.0.97.3:35005): tablet 8d4b56f62a8747619375b1f3a197ce92 (table test-table [id=4f4c1ac5d1264978907ac936c8c3a0b0]) successfully deleted
I20260927 22:08:52.813349 2730 ts_tablet_manager.cc:1918] T 8d4b56f62a8747619375b1f3a197ce92 P 6c310f6d55dc4eabbea8f30ca8ced5e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:08:52.813535 2730 log.cc:1196] T 8d4b56f62a8747619375b1f3a197ce92 P 6c310f6d55dc4eabbea8f30ca8ced5e8: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/8d4b56f62a8747619375b1f3a197ce92
I20260927 22:08:52.813886 2740 tablet_bootstrap.cc:492] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29: Bootstrap starting.
I20260927 22:08:52.813871 2128 catalog_manager.cc:4841] TS 02c9b6626fc84e0281183902da419c29 (127.0.97.1:33801): tablet 8d4b56f62a8747619375b1f3a197ce92 (table test-table [id=4f4c1ac5d1264978907ac936c8c3a0b0]) successfully deleted
I20260927 22:08:52.814013 2730 ts_tablet_manager.cc:1939] T 8d4b56f62a8747619375b1f3a197ce92 P 6c310f6d55dc4eabbea8f30ca8ced5e8: Deleting consensus metadata
I20260927 22:08:52.816012 2740 tablet_bootstrap.cc:654] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:52.816359 2742 tablet_bootstrap.cc:492] T 52bf5fd2b97449dbb00092b1e5a7f399 P 98cc05014da04d4d9bdd2e29d9d24135: Bootstrap starting.
I20260927 22:08:52.817867 2741 tablet_bootstrap.cc:492] T 52bf5fd2b97449dbb00092b1e5a7f399 P 6c310f6d55dc4eabbea8f30ca8ced5e8: Bootstrap starting.
I20260927 22:08:52.817883 2742 tablet_bootstrap.cc:654] T 52bf5fd2b97449dbb00092b1e5a7f399 P 98cc05014da04d4d9bdd2e29d9d24135: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:52.818943 2740 tablet_bootstrap.cc:492] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29: No bootstrap required, opened a new log
I20260927 22:08:52.819136 2740 ts_tablet_manager.cc:1397] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260927 22:08:52.819504 2741 tablet_bootstrap.cc:654] T 52bf5fd2b97449dbb00092b1e5a7f399 P 6c310f6d55dc4eabbea8f30ca8ced5e8: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:52.819901 2740 raft_consensus.cc:348] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:52.820178 2740 raft_consensus.cc:374] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:52.820283 2740 raft_consensus.cc:729] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c9b6626fc84e0281183902da419c29, State: Initialized, Role: FOLLOWER
I20260927 22:08:52.820590 2740 consensus_queue.cc:260] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:52.821156 2740 ts_tablet_manager.cc:1428] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 22:08:52.824935 2741 tablet_bootstrap.cc:492] T 52bf5fd2b97449dbb00092b1e5a7f399 P 6c310f6d55dc4eabbea8f30ca8ced5e8: No bootstrap required, opened a new log
I20260927 22:08:52.825145 2741 ts_tablet_manager.cc:1397] T 52bf5fd2b97449dbb00092b1e5a7f399 P 6c310f6d55dc4eabbea8f30ca8ced5e8: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.001s
I20260927 22:08:52.825239 2742 tablet_bootstrap.cc:492] T 52bf5fd2b97449dbb00092b1e5a7f399 P 98cc05014da04d4d9bdd2e29d9d24135: No bootstrap required, opened a new log
I20260927 22:08:52.825412 2742 ts_tablet_manager.cc:1397] T 52bf5fd2b97449dbb00092b1e5a7f399 P 98cc05014da04d4d9bdd2e29d9d24135: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20260927 22:08:52.825958 2741 raft_consensus.cc:348] T 52bf5fd2b97449dbb00092b1e5a7f399 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:52.826206 2741 raft_consensus.cc:374] T 52bf5fd2b97449dbb00092b1e5a7f399 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:52.826118 2742 raft_consensus.cc:348] T 52bf5fd2b97449dbb00092b1e5a7f399 P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:52.826287 2741 raft_consensus.cc:729] T 52bf5fd2b97449dbb00092b1e5a7f399 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c310f6d55dc4eabbea8f30ca8ced5e8, State: Initialized, Role: FOLLOWER
I20260927 22:08:52.826373 2742 raft_consensus.cc:374] T 52bf5fd2b97449dbb00092b1e5a7f399 P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:52.826464 2742 raft_consensus.cc:729] T 52bf5fd2b97449dbb00092b1e5a7f399 P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98cc05014da04d4d9bdd2e29d9d24135, State: Initialized, Role: FOLLOWER
I20260927 22:08:52.826520 2741 consensus_queue.cc:260] T 52bf5fd2b97449dbb00092b1e5a7f399 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:52.826738 2742 consensus_queue.cc:260] T 52bf5fd2b97449dbb00092b1e5a7f399 P 98cc05014da04d4d9bdd2e29d9d24135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:52.827241 2742 ts_tablet_manager.cc:1428] T 52bf5fd2b97449dbb00092b1e5a7f399 P 98cc05014da04d4d9bdd2e29d9d24135: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:08:52.827854 2741 ts_tablet_manager.cc:1428] T 52bf5fd2b97449dbb00092b1e5a7f399 P 6c310f6d55dc4eabbea8f30ca8ced5e8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 22:08:52.905614 2126 catalog_manager.cc:4841] TS 6c310f6d55dc4eabbea8f30ca8ced5e8 (127.0.97.2:38195): tablet 8d4b56f62a8747619375b1f3a197ce92 (table test-table [id=4f4c1ac5d1264978907ac936c8c3a0b0]) successfully deleted
I20260927 22:08:52.912110 2708 raft_consensus.cc:482] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:08:52.912317 2708 raft_consensus.cc:504] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:52.913046 2708 leader_election.cc:288] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c310f6d55dc4eabbea8f30ca8ced5e8 (127.0.97.2:38195), 98cc05014da04d4d9bdd2e29d9d24135 (127.0.97.3:35005)
I20260927 22:08:52.913472 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52bf5fd2b97449dbb00092b1e5a7f399" candidate_uuid: "02c9b6626fc84e0281183902da419c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" is_pre_election: true
I20260927 22:08:52.913692 2437 raft_consensus.cc:2454] T 52bf5fd2b97449dbb00092b1e5a7f399 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c9b6626fc84e0281183902da419c29 in term 0.
I20260927 22:08:52.913584 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52bf5fd2b97449dbb00092b1e5a7f399" candidate_uuid: "02c9b6626fc84e0281183902da419c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98cc05014da04d4d9bdd2e29d9d24135" is_pre_election: true
I20260927 22:08:52.913868 2576 raft_consensus.cc:2454] T 52bf5fd2b97449dbb00092b1e5a7f399 P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c9b6626fc84e0281183902da419c29 in term 0.
I20260927 22:08:52.914098 2229 leader_election.cc:302] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c9b6626fc84e0281183902da419c29, 6c310f6d55dc4eabbea8f30ca8ced5e8; no voters:
I20260927 22:08:52.914335 2708 raft_consensus.cc:2781] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:08:52.914451 2708 raft_consensus.cc:482] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:08:52.914551 2708 raft_consensus.cc:3037] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:53.000341 2708 raft_consensus.cc:504] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:53.001065 2708 leader_election.cc:288] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29 [CANDIDATE]: Term 1 election: Requested vote from peers 6c310f6d55dc4eabbea8f30ca8ced5e8 (127.0.97.2:38195), 98cc05014da04d4d9bdd2e29d9d24135 (127.0.97.3:35005)
I20260927 22:08:53.001664 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52bf5fd2b97449dbb00092b1e5a7f399" candidate_uuid: "02c9b6626fc84e0281183902da419c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98cc05014da04d4d9bdd2e29d9d24135"
I20260927 22:08:53.001709 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52bf5fd2b97449dbb00092b1e5a7f399" candidate_uuid: "02c9b6626fc84e0281183902da419c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8"
I20260927 22:08:53.001873 2576 raft_consensus.cc:3037] T 52bf5fd2b97449dbb00092b1e5a7f399 P 98cc05014da04d4d9bdd2e29d9d24135 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:53.001986 2437 raft_consensus.cc:3037] T 52bf5fd2b97449dbb00092b1e5a7f399 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:53.004300 2576 raft_consensus.cc:2454] T 52bf5fd2b97449dbb00092b1e5a7f399 P 98cc05014da04d4d9bdd2e29d9d24135 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c9b6626fc84e0281183902da419c29 in term 1.
I20260927 22:08:53.004797 2232 leader_election.cc:302] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c9b6626fc84e0281183902da419c29, 98cc05014da04d4d9bdd2e29d9d24135; no voters:
I20260927 22:08:53.005088 2708 raft_consensus.cc:2781] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:08:53.005230 2708 raft_consensus.cc:686] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29 [term 1 LEADER]: Becoming Leader. State: Replica: 02c9b6626fc84e0281183902da419c29, State: Running, Role: LEADER
I20260927 22:08:53.005363 2437 raft_consensus.cc:2454] T 52bf5fd2b97449dbb00092b1e5a7f399 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c9b6626fc84e0281183902da419c29 in term 1.
I20260927 22:08:53.005577 2708 consensus_queue.cc:237] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } }
I20260927 22:08:53.008509 2141 catalog_manager.cc:5495] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c9b6626fc84e0281183902da419c29 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "02c9b6626fc84e0281183902da419c29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c9b6626fc84e0281183902da419c29" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 } health_report { overall_health: UNKNOWN } } }
I20260927 22:08:53.040655 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:53.035968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44438 (local address 127.0.97.1:33801)
0927 22:08:53.036466 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:53.036470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:53.036489 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:08:53.036534 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:53.036537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:53.036607 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:53.036686 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:08:53.037073 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:53.037810 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:53.039071 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:53.039465 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:53.039558 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:53.039876 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:08:53.040048 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:08:53.040135 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:53.040248 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 22:08:53.040334 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":286,"threads_started":1}
I20260927 22:08:53.053854 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:53.048570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34487 (local address 127.0.97.1:33801)
0927 22:08:53.048733 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:53.048737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:53.049123 (+ 386us) server_negotiation.cc:407] Connection header received
0927 22:08:53.049277 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:53.049280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:53.049338 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:53.049444 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:08:53.050259 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:53.051220 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:53.052465 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:53.052824 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:53.052913 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:53.053260 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:08:53.053385 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:08:53.053479 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:53.053555 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 22:08:53.053618 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":68}
I20260927 22:08:53.053915 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:53.048409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33801 (local address 127.0.97.1:34487)
0927 22:08:53.048993 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:53.049008 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:53.049178 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:53.049462 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:53.049465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:53.049473 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:08:53.050110 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:53.050118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:53.051319 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:53.051323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:53.052330 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:53.052339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:53.052954 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:53.052958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:53.053089 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:53.053121 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:08:53.053464 (+ 343us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:08:53.053472 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:08:53.053528 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 22:08:53.053629 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":3,"thread_start_us":333,"threads_started":1}
I20260927 22:08:53.061043 2576 raft_consensus.cc:1261] T 52bf5fd2b97449dbb00092b1e5a7f399 P 98cc05014da04d4d9bdd2e29d9d24135 [term 1 FOLLOWER]: Refusing update from remote peer 02c9b6626fc84e0281183902da419c29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:08:53.061110 2437 raft_consensus.cc:1261] T 52bf5fd2b97449dbb00092b1e5a7f399 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 1 FOLLOWER]: Refusing update from remote peer 02c9b6626fc84e0281183902da419c29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:08:53.061597 2652 consensus_queue.cc:1035] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c310f6d55dc4eabbea8f30ca8ced5e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:08:53.061849 2708 consensus_queue.cc:1035] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98cc05014da04d4d9bdd2e29d9d24135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:08:53.085542 2706 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 22:08:54.676082 2142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:08:54.676295 2142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:08:54.680708 2142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4bc9c4b60ab0473a9677192741a90068: Sending DeleteTablet for 3 replicas of tablet 52bf5fd2b97449dbb00092b1e5a7f399
I20260927 22:08:54.681664 2282 tablet_service.cc:1514] Processing DeleteTablet for tablet 52bf5fd2b97449dbb00092b1e5a7f399 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:08:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44416
I20260927 22:08:54.681809 2417 tablet_service.cc:1514] Processing DeleteTablet for tablet 52bf5fd2b97449dbb00092b1e5a7f399 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:08:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48192
I20260927 22:08:54.681958 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet 52bf5fd2b97449dbb00092b1e5a7f399 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:08:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47216
I20260927 22:08:54.682509 2785 tablet_replica.cc:330] T 52bf5fd2b97449dbb00092b1e5a7f399 P 6c310f6d55dc4eabbea8f30ca8ced5e8: stopping tablet replica
I20260927 22:08:54.682863 2785 raft_consensus.cc:2229] T 52bf5fd2b97449dbb00092b1e5a7f399 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:08:54.683220 2785 raft_consensus.cc:2258] T 52bf5fd2b97449dbb00092b1e5a7f399 P 6c310f6d55dc4eabbea8f30ca8ced5e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:08:54.683302 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 2213
I20260927 22:08:54.684407 2785 ts_tablet_manager.cc:1905] T 52bf5fd2b97449dbb00092b1e5a7f399 P 6c310f6d55dc4eabbea8f30ca8ced5e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:08:54.694936 2785 ts_tablet_manager.cc:1918] T 52bf5fd2b97449dbb00092b1e5a7f399 P 6c310f6d55dc4eabbea8f30ca8ced5e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:08:54.695240 2785 log.cc:1196] T 52bf5fd2b97449dbb00092b1e5a7f399 P 6c310f6d55dc4eabbea8f30ca8ced5e8: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/52bf5fd2b97449dbb00092b1e5a7f399
I20260927 22:08:54.695586 2784 tablet_replica.cc:330] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29: stopping tablet replica
I20260927 22:08:54.695849 2784 raft_consensus.cc:2229] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:08:54.696072 2785 ts_tablet_manager.cc:1939] T 52bf5fd2b97449dbb00092b1e5a7f399 P 6c310f6d55dc4eabbea8f30ca8ced5e8: Deleting consensus metadata
I20260927 22:08:54.696467 2784 raft_consensus.cc:2258] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:08:54.697491 2784 ts_tablet_manager.cc:1905] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:08:54.697448 2126 catalog_manager.cc:4841] TS 6c310f6d55dc4eabbea8f30ca8ced5e8 (127.0.97.2:38195): tablet 52bf5fd2b97449dbb00092b1e5a7f399 (table test-table [id=f2f66c81a4e7424a948f56340e3bb234]) successfully deleted
I20260927 22:08:54.697733 2786 tablet_replica.cc:330] T 52bf5fd2b97449dbb00092b1e5a7f399 P 98cc05014da04d4d9bdd2e29d9d24135: stopping tablet replica
I20260927 22:08:54.697921 2786 raft_consensus.cc:2229] T 52bf5fd2b97449dbb00092b1e5a7f399 P 98cc05014da04d4d9bdd2e29d9d24135 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:08:54.698167 2786 raft_consensus.cc:2258] T 52bf5fd2b97449dbb00092b1e5a7f399 P 98cc05014da04d4d9bdd2e29d9d24135 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:08:54.699605 2786 ts_tablet_manager.cc:1905] T 52bf5fd2b97449dbb00092b1e5a7f399 P 98cc05014da04d4d9bdd2e29d9d24135: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:08:54.703867 2786 ts_tablet_manager.cc:1918] T 52bf5fd2b97449dbb00092b1e5a7f399 P 98cc05014da04d4d9bdd2e29d9d24135: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:08:54.704001 2786 log.cc:1196] T 52bf5fd2b97449dbb00092b1e5a7f399 P 98cc05014da04d4d9bdd2e29d9d24135: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/52bf5fd2b97449dbb00092b1e5a7f399
I20260927 22:08:54.704455 2786 ts_tablet_manager.cc:1939] T 52bf5fd2b97449dbb00092b1e5a7f399 P 98cc05014da04d4d9bdd2e29d9d24135: Deleting consensus metadata
I20260927 22:08:54.705351 2129 catalog_manager.cc:4841] TS 98cc05014da04d4d9bdd2e29d9d24135 (127.0.97.3:35005): tablet 52bf5fd2b97449dbb00092b1e5a7f399 (table test-table [id=f2f66c81a4e7424a948f56340e3bb234]) successfully deleted
I20260927 22:08:54.705799 2784 ts_tablet_manager.cc:1918] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:08:54.705948 2784 log.cc:1196] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546912983837-388-0/minicluster-data/ts-0/wal/wals/52bf5fd2b97449dbb00092b1e5a7f399
I20260927 22:08:54.706499 2784 ts_tablet_manager.cc:1939] T 52bf5fd2b97449dbb00092b1e5a7f399 P 02c9b6626fc84e0281183902da419c29: Deleting consensus metadata
I20260927 22:08:54.707707 2128 catalog_manager.cc:4841] TS 02c9b6626fc84e0281183902da419c29 (127.0.97.1:33801): tablet 52bf5fd2b97449dbb00092b1e5a7f399 (table test-table [id=f2f66c81a4e7424a948f56340e3bb234]) successfully deleted
Sep 27 22:08:54 dist-test-slave-mp03 krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546932, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:08:54.730876 2788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:54.685400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44450 (local address 127.0.97.1:33801)
0927 22:08:54.709599 (+ 24199us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:54.709603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:54.709621 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:08:54.709661 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:54.709664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:54.709720 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:54.709803 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:54.711083 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:54.711777 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:54.712936 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:54.713346 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:54.713442 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:54.725326 (+ 11884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:54.725343 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:54.725362 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:54.725403 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:54.728321 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:54.729024 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:54.729031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:54.729034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:54.729162 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:54.729507 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:54.729510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:54.729512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:54.729857 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:54.730053 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:54.730355 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 22:08:54.730519 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24086,"thread_start_us":24640,"threads_started":1}
W20260927 22:08:54.731520 2344 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44450
W20260927 22:08:54.732090 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 2213: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:08:54.732278 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 2213
I20260927 22:08:54.759542 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 2353
Sep 27 22:08:54 dist-test-slave-mp03 krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546932, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:08:54.775586 2789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:54.760279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48194 (local address 127.0.97.2:38195)
0927 22:08:54.760914 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:54.760919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:54.760933 (+ 14us) server_negotiation.cc:407] Connection header received
0927 22:08:54.760984 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:54.760988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:54.761042 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:54.761163 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:54.762381 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:54.763413 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:54.764787 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:54.765200 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:54.765325 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:54.769702 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:54.769739 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:54.769755 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:54.769790 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:54.772913 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:54.773540 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:54.773548 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:54.773551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:54.773765 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:54.774120 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:54.774123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:54.774125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:54.774439 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:54.774705 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:54.775172 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 22:08:54.775331 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":420,"threads_started":1}
W20260927 22:08:54.776238 2483 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48194
W20260927 22:08:54.776853 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 2353: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:08:54.777015 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 2353
I20260927 22:08:54.794898 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 2492
Sep 27 22:08:54 dist-test-slave-mp03 krb5kdc[2097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546932, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:08:54.813110 2790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:54.795644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.0.97.3:35005)
0927 22:08:54.796214 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:54.796219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:54.796234 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:08:54.796281 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:54.796284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:54.796338 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:54.796423 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:54.798058 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:54.798905 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:54.800652 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:54.801081 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:54.801174 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:54.806385 (+ 5211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:54.806403 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:54.806425 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:54.806470 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:54.810159 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:54.810861 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:54.810870 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:54.810874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:54.811041 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:54.811391 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:54.811396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:54.811400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:54.811826 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:54.812118 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:54.812460 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 22:08:54.812691 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":300,"threads_started":1}
W20260927 22:08:54.813860 2622 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47250
W20260927 22:08:54.814538 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 2492: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:08:54.814788 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 2492
I20260927 22:08:54.836308 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 2110
I20260927 22:08:54.837419 2182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:08:54.966946 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 2110
2026-09-27T22:08:54Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8553 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:08:55 dist-test-slave-mp03 krb5kdc[2794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:08:55 dist-test-slave-mp03 krb5kdc[2794](info): set up 2 sockets
Sep 27 22:08:55 dist-test-slave-mp03 krb5kdc[2794](info): commencing operation
krb5kdc: starting...
W20260927 22:08:56.054857 388 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:08:56 dist-test-slave-mp03 krb5kdc[2794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:08:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:08:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:08:56.242934 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37251
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:08:56.557754 2807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:08:56.558111 2807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:08:56.558264 2807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:08:56.565490 2807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:08:56.565636 2807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:08:56.565745 2807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:08:56.565856 2807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:08:56.565943 2807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:08:56.576418 2807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:08:56.578022 2807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:08:56.579591 2807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:08:56.593194 2812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:56.593264 2813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:56.593298 2815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:08:56.594630 2807 server_base.cc:1034] running on GCE node
I20260927 22:08:56.595553 2807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:08:56.597220 2807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:08:56.598491 2807 hybrid_clock.cc:648] HybridClock initialized: now 1790546936598445 us; error 61 us; skew 500 ppm
I20260927 22:08:56.598872 2807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:08:56 dist-test-slave-mp03 krb5kdc[2794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:08:56.602015 2807 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:08:56.603449 2807 webserver.cc:460] Webserver started at http://127.0.97.62:41019/ using document root <none> and password file <none>
I20260927 22:08:56.604082 2807 fs_manager.cc:359] Metadata directory not provided
I20260927 22:08:56.604176 2807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:08:56.604426 2807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:08:56.607486 2807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "93476085937d4f36880d0fb372e8b9e7"
format_stamp: "Formatted at 2026-09-27 22:08:56 on dist-test-slave-mp03"
server_key: "7c378a7d17e7779bfce572b1ccbe4adf"
server_key_iv: "c12267e09ad0ca43aa0c1cd44c2774a9"
server_key_version: "encryptionkey@0"
I20260927 22:08:56.608343 2807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "93476085937d4f36880d0fb372e8b9e7"
format_stamp: "Formatted at 2026-09-27 22:08:56 on dist-test-slave-mp03"
server_key: "7c378a7d17e7779bfce572b1ccbe4adf"
server_key_iv: "c12267e09ad0ca43aa0c1cd44c2774a9"
server_key_version: "encryptionkey@0"
I20260927 22:08:56.613701 2807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:08:56.617274 2822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:56.618669 2807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:08:56.618875 2807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "93476085937d4f36880d0fb372e8b9e7"
format_stamp: "Formatted at 2026-09-27 22:08:56 on dist-test-slave-mp03"
server_key: "7c378a7d17e7779bfce572b1ccbe4adf"
server_key_iv: "c12267e09ad0ca43aa0c1cd44c2774a9"
server_key_version: "encryptionkey@0"
I20260927 22:08:56.619095 2807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:08:56.634388 2807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:08:56.638775 2807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:08:56.639057 2807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:08:56.656736 2807 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37251
I20260927 22:08:56.656766 2873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37251 every 8 connection(s)
I20260927 22:08:56.658143 2807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:08:56.659251 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 2807
I20260927 22:08:56.659471 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:08:56.660259 388 external_mini_cluster.cc:1428] Setting key 561da0573dcd5db1d6cf589be69460f5
I20260927 22:08:56.662889 2874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:08:56 dist-test-slave-mp03 krb5kdc[2794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:08:56.675839 2874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93476085937d4f36880d0fb372e8b9e7: Bootstrap starting.
I20260927 22:08:56.680086 2874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93476085937d4f36880d0fb372e8b9e7: Neither blocks nor log segments found. Creating new log.
I20260927 22:08:56.681383 2874 log.cc:824] T 00000000000000000000000000000000 P 93476085937d4f36880d0fb372e8b9e7: Log is configured to *not* fsync() on all Append() calls
I20260927 22:08:56.683646 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:56.662305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56720 (local address 127.0.97.62:37251)
0927 22:08:56.663092 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:56.663104 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:56.663143 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:08:56.664686 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:56.664705 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:56.665063 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:56.665390 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:56.667178 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:56.668546 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:56.670285 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:56.671043 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:56.671178 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:56.675672 (+ 4494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:56.675682 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:56.675701 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:56.675743 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:56.679545 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:56.680209 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:56.680218 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:56.680227 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:56.680436 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:56.680786 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:56.680790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:56.680793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:56.681518 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:56.681777 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:56.682199 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 22:08:56.682675 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":382,"threads_started":1}
I20260927 22:08:56.684393 2874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93476085937d4f36880d0fb372e8b9e7: No bootstrap required, opened a new log
I20260927 22:08:56.691571 2874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93476085937d4f36880d0fb372e8b9e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93476085937d4f36880d0fb372e8b9e7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } }
I20260927 22:08:56.691962 2874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93476085937d4f36880d0fb372e8b9e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:08:56.692102 2874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93476085937d4f36880d0fb372e8b9e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93476085937d4f36880d0fb372e8b9e7, State: Initialized, Role: FOLLOWER
I20260927 22:08:56.692706 2874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93476085937d4f36880d0fb372e8b9e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93476085937d4f36880d0fb372e8b9e7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } }
I20260927 22:08:56.692955 2874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93476085937d4f36880d0fb372e8b9e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:08:56.693086 2874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93476085937d4f36880d0fb372e8b9e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:08:56.693270 2874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93476085937d4f36880d0fb372e8b9e7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:08:56.695467 2874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93476085937d4f36880d0fb372e8b9e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93476085937d4f36880d0fb372e8b9e7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } }
I20260927 22:08:56.695963 2874 leader_election.cc:302] T 00000000000000000000000000000000 P 93476085937d4f36880d0fb372e8b9e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93476085937d4f36880d0fb372e8b9e7; no voters:
I20260927 22:08:56.696524 2874 leader_election.cc:288] T 00000000000000000000000000000000 P 93476085937d4f36880d0fb372e8b9e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:08:56.696669 2879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93476085937d4f36880d0fb372e8b9e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:08:56.697679 2879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93476085937d4f36880d0fb372e8b9e7 [term 1 LEADER]: Becoming Leader. State: Replica: 93476085937d4f36880d0fb372e8b9e7, State: Running, Role: LEADER
I20260927 22:08:56.698230 2879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93476085937d4f36880d0fb372e8b9e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93476085937d4f36880d0fb372e8b9e7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } }
I20260927 22:08:56.698726 2874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93476085937d4f36880d0fb372e8b9e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:08:56.700996 2881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93476085937d4f36880d0fb372e8b9e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93476085937d4f36880d0fb372e8b9e7. Latest consensus state: current_term: 1 leader_uuid: "93476085937d4f36880d0fb372e8b9e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93476085937d4f36880d0fb372e8b9e7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } } }
I20260927 22:08:56.701426 2881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93476085937d4f36880d0fb372e8b9e7 [sys.catalog]: This master's current role is: LEADER
I20260927 22:08:56.702010 2880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93476085937d4f36880d0fb372e8b9e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93476085937d4f36880d0fb372e8b9e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93476085937d4f36880d0fb372e8b9e7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37251 } } }
I20260927 22:08:56.702281 2880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93476085937d4f36880d0fb372e8b9e7 [sys.catalog]: This master's current role is: LEADER
I20260927 22:08:56.702316 2888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:08:56.708501 2888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:08:56.716914 2888 catalog_manager.cc:1347] Generated new cluster ID: 485c0e2ed11942459792acacd54d9af4
I20260927 22:08:56.717033 2888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:08:56.735736 2888 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:08:56.737674 2888 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:08:56.747601 2888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93476085937d4f36880d0fb372e8b9e7: Generated new TSK 0
I20260927 22:08:56.748658 2888 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:08:56.792362 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37251
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:08:57.134361 2900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:08:57.134724 2900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:08:57.134853 2900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:08:57.142162 2900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:08:57.142323 2900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:08:57.142511 2900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:08:57.152951 2900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:08:57.154206 2900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:08:57.155475 2900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:08:57.168007 2905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:57.168088 2906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:57.168088 2908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:08:57.168962 2900 server_base.cc:1034] running on GCE node
I20260927 22:08:57.169611 2900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:08:57.170553 2900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:08:57.171805 2900 hybrid_clock.cc:648] HybridClock initialized: now 1790546937171745 us; error 54 us; skew 500 ppm
I20260927 22:08:57.172130 2900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:08:57 dist-test-slave-mp03 krb5kdc[2794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:08:57.175218 2900 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:08:57.177021 2900 webserver.cc:460] Webserver started at http://127.0.97.1:43261/ using document root <none> and password file <none>
I20260927 22:08:57.177995 2900 fs_manager.cc:359] Metadata directory not provided
I20260927 22:08:57.178143 2900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:08:57.178479 2900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:08:57.181603 2900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "46a9e6f8cb5a482d916e2cc85a0cd2ae"
format_stamp: "Formatted at 2026-09-27 22:08:57 on dist-test-slave-mp03"
server_key: "88dda7c5f6f1abe4dbad055c39cc7b7d"
server_key_iv: "6d334d8fba535a636e980dc10de3f2ca"
server_key_version: "encryptionkey@0"
I20260927 22:08:57.182318 2900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "46a9e6f8cb5a482d916e2cc85a0cd2ae"
format_stamp: "Formatted at 2026-09-27 22:08:57 on dist-test-slave-mp03"
server_key: "88dda7c5f6f1abe4dbad055c39cc7b7d"
server_key_iv: "6d334d8fba535a636e980dc10de3f2ca"
server_key_version: "encryptionkey@0"
I20260927 22:08:57.187414 2900 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:08:57.190687 2915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:57.191954 2900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:08:57.192131 2900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "46a9e6f8cb5a482d916e2cc85a0cd2ae"
format_stamp: "Formatted at 2026-09-27 22:08:57 on dist-test-slave-mp03"
server_key: "88dda7c5f6f1abe4dbad055c39cc7b7d"
server_key_iv: "6d334d8fba535a636e980dc10de3f2ca"
server_key_version: "encryptionkey@0"
I20260927 22:08:57.192330 2900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:08:57.204588 2900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:08:57.208484 2900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:08:57.208758 2900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:08:57.209707 2900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:08:57.211062 2900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:08:57.211158 2900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:57.211303 2900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:08:57.211374 2900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:57.242976 2900 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38705
I20260927 22:08:57.243024 3027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38705 every 8 connection(s)
I20260927 22:08:57.244256 2900 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:08:57.251113 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 2900
I20260927 22:08:57.251320 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:08:57.251909 388 external_mini_cluster.cc:1428] Setting key a2f78defdcdb81cef1872f7613e65157
Sep 27 22:08:57 dist-test-slave-mp03 krb5kdc[2794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:08:57.265398 3030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:57.246762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34165 (local address 127.0.97.62:37251)
0927 22:08:57.247163 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:57.247173 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:57.248378 (+ 1205us) server_negotiation.cc:407] Connection header received
0927 22:08:57.250206 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:57.250211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:57.250297 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:57.250416 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:57.252096 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:57.252975 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:57.255260 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:57.255644 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:57.255742 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:57.259933 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:57.259938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:57.259941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:57.259970 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:57.262442 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:57.263177 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:57.263187 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:57.263190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:57.263301 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:57.263720 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:57.263725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:57.263729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:57.263991 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:57.264150 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:57.264818 (+ 668us) server_negotiation.cc:299] Negotiation successful
0927 22:08:57.265039 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":184,"threads_started":1}
I20260927 22:08:57.266345 3031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:57.247380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37251 (local address 127.0.97.1:34165)
0927 22:08:57.248212 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:57.248263 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:57.249909 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:57.250701 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:57.250709 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:57.251294 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:08:57.251899 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:57.251916 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:57.253152 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:57.253156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:57.255113 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:57.255131 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:57.255872 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:57.255882 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:57.256223 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:57.256975 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:08:57.257012 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:08:57.259702 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:08:57.262581 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:57.262590 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:57.262627 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:57.263043 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:57.263407 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:57.263413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:57.263417 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:57.263603 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:57.264123 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:08:57.264131 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:08:57.264459 (+ 328us) client_negotiation.cc:769] Sending connection context
0927 22:08:57.264726 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 22:08:57.265251 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":345,"threads_started":1}
I20260927 22:08:57.267892 3028 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37251
I20260927 22:08:57.268214 3028 heartbeater.cc:464] Registering TS with master...
I20260927 22:08:57.268884 3028 heartbeater.cc:511] Master 127.0.97.62:37251 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:08:57.271070 2839 ts_manager.cc:194] Registered new tserver with Master: 46a9e6f8cb5a482d916e2cc85a0cd2ae (127.0.97.1:38705)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 22:08:57.272644 2839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:08:57.294335 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37251
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:08:57.621794 3034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:08:57.622097 3034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:08:57.622249 3034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:08:57.629981 3034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:08:57.630096 3034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:08:57.630268 3034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:08:57.640241 3034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:08:57.641644 3034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:08:57.643028 3034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:08:57.655829 3039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:57.656005 3042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:57.655974 3040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:08:57.659865 3034 server_base.cc:1034] running on GCE node
I20260927 22:08:57.660480 3034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:08:57.661513 3034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:08:57.662698 3034 hybrid_clock.cc:648] HybridClock initialized: now 1790546937662646 us; error 45 us; skew 500 ppm
I20260927 22:08:57.663010 3034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:08:57 dist-test-slave-mp03 krb5kdc[2794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:08:57.666050 3034 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:08:57.667595 3034 webserver.cc:460] Webserver started at http://127.0.97.2:38833/ using document root <none> and password file <none>
I20260927 22:08:57.668267 3034 fs_manager.cc:359] Metadata directory not provided
I20260927 22:08:57.668387 3034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:08:57.668697 3034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:08:57.671537 3034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "eba19ace617e4c11b01da38d69182654"
format_stamp: "Formatted at 2026-09-27 22:08:57 on dist-test-slave-mp03"
server_key: "2809032b42566eaae9829b5361a2072b"
server_key_iv: "051a5484039fe6547e3618bbf5bbd660"
server_key_version: "encryptionkey@0"
I20260927 22:08:57.672283 3034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "eba19ace617e4c11b01da38d69182654"
format_stamp: "Formatted at 2026-09-27 22:08:57 on dist-test-slave-mp03"
server_key: "2809032b42566eaae9829b5361a2072b"
server_key_iv: "051a5484039fe6547e3618bbf5bbd660"
server_key_version: "encryptionkey@0"
I20260927 22:08:57.677242 3034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:08:57.680538 3049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:57.681771 3034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:08:57.681922 3034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "eba19ace617e4c11b01da38d69182654"
format_stamp: "Formatted at 2026-09-27 22:08:57 on dist-test-slave-mp03"
server_key: "2809032b42566eaae9829b5361a2072b"
server_key_iv: "051a5484039fe6547e3618bbf5bbd660"
server_key_version: "encryptionkey@0"
I20260927 22:08:57.682119 3034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:08:57.697235 3034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:08:57.701718 3034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:08:57.701992 3034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:08:57.703068 3034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:08:57.704408 3034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:08:57.704478 3034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:57.704650 3034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:08:57.704701 3034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:57.737625 3034 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34973
I20260927 22:08:57.737704 3161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34973 every 8 connection(s)
I20260927 22:08:57.738981 3034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:08:57.742964 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 3034
I20260927 22:08:57.743147 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:08:57.743685 388 external_mini_cluster.cc:1428] Setting key 02232901687c4480c3a8b1794b882d01
Sep 27 22:08:57 dist-test-slave-mp03 krb5kdc[2794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:08:57.762766 3030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:57.741518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40101 (local address 127.0.97.62:37251)
0927 22:08:57.741689 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:57.741693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:57.742917 (+ 1224us) server_negotiation.cc:407] Connection header received
0927 22:08:57.745194 (+ 2277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:57.745198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:57.745270 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:57.745376 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:57.747548 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:57.748330 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:57.750856 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:57.751276 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:57.751400 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:57.755460 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:57.755469 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:57.755477 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:57.755523 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:57.759283 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:57.760193 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:57.760199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:57.760203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:57.760361 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:57.760777 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:57.760785 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:57.760791 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:57.761193 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:57.761351 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:57.762049 (+ 698us) server_negotiation.cc:299] Negotiation successful
0927 22:08:57.762331 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:08:57.763518 3164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:57.741890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37251 (local address 127.0.97.2:40101)
0927 22:08:57.742703 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:57.742764 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:57.744923 (+ 2159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:57.745645 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:57.745658 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:57.746502 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:08:57.747372 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:57.747399 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:57.748516 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:57.748524 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:57.750727 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:57.750743 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:57.751502 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:57.751509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:57.751739 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:57.752445 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:08:57.752482 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:08:57.755202 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:08:57.759425 (+ 4223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:57.759433 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:57.759449 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:57.760019 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:57.760478 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:57.760482 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:57.760484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:57.760655 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:57.761326 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:08:57.761331 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:08:57.761677 (+ 346us) client_negotiation.cc:769] Sending connection context
0927 22:08:57.761901 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 22:08:57.762305 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":258,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:08:57.764995 3162 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37251
I20260927 22:08:57.765429 3162 heartbeater.cc:464] Registering TS with master...
I20260927 22:08:57.766273 3162 heartbeater.cc:511] Master 127.0.97.62:37251 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 22:08:57.768122 2839 ts_manager.cc:194] Registered new tserver with Master: eba19ace617e4c11b01da38d69182654 (127.0.97.2:34973)
I20260927 22:08:57.768999 2839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40101
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:08:57.790586 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37251
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:08:58.114908 3167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:08:58.115197 3167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:08:58.115372 3167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:08:58.123291 3167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:08:58.123448 3167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:08:58.123716 3167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:08:58.134716 3167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:08:58.136361 3167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:08:58.137996 3167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:08:58.151147 3172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:58.151196 3175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:08:58.151093 3173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:08:58.152005 3167 server_base.cc:1034] running on GCE node
I20260927 22:08:58.152604 3167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:08:58.153517 3167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:08:58.154711 3167 hybrid_clock.cc:648] HybridClock initialized: now 1790546938154653 us; error 40 us; skew 500 ppm
I20260927 22:08:58.155052 3167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:08:58 dist-test-slave-mp03 krb5kdc[2794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:08:58.158277 3167 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:08:58.159828 3167 webserver.cc:460] Webserver started at http://127.0.97.3:45495/ using document root <none> and password file <none>
I20260927 22:08:58.160540 3167 fs_manager.cc:359] Metadata directory not provided
I20260927 22:08:58.160698 3167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:08:58.161070 3167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:08:58.164181 3167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "e97ec10b059f4373b1f26dd0197af4d5"
format_stamp: "Formatted at 2026-09-27 22:08:58 on dist-test-slave-mp03"
server_key: "7d1b38531e83bc71dc9f22683a19f6e2"
server_key_iv: "cf357a76f64ced86abcb5834eab42b90"
server_key_version: "encryptionkey@0"
I20260927 22:08:58.164959 3167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e97ec10b059f4373b1f26dd0197af4d5"
format_stamp: "Formatted at 2026-09-27 22:08:58 on dist-test-slave-mp03"
server_key: "7d1b38531e83bc71dc9f22683a19f6e2"
server_key_iv: "cf357a76f64ced86abcb5834eab42b90"
server_key_version: "encryptionkey@0"
I20260927 22:08:58.171661 3167 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 22:08:58.175217 3182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:58.176515 3167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:08:58.176684 3167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "e97ec10b059f4373b1f26dd0197af4d5"
format_stamp: "Formatted at 2026-09-27 22:08:58 on dist-test-slave-mp03"
server_key: "7d1b38531e83bc71dc9f22683a19f6e2"
server_key_iv: "cf357a76f64ced86abcb5834eab42b90"
server_key_version: "encryptionkey@0"
I20260927 22:08:58.176880 3167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:08:58.192162 3167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:08:58.196532 3167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:08:58.196775 3167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:08:58.197724 3167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:08:58.199069 3167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:08:58.199172 3167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:58.199282 3167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:08:58.199328 3167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:08:58.232349 3167 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46279
I20260927 22:08:58.232394 3294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46279 every 8 connection(s)
I20260927 22:08:58.233652 3167 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:08:58.238119 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 3167
I20260927 22:08:58.238318 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:08:58.238765 388 external_mini_cluster.cc:1428] Setting key 5731127934a9965bf6b508421033dcc8
Sep 27 22:08:58 dist-test-slave-mp03 krb5kdc[2794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:08:58.253197 3030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:58.236326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54839 (local address 127.0.97.62:37251)
0927 22:08:58.236490 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:58.236494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:58.237675 (+ 1181us) server_negotiation.cc:407] Connection header received
0927 22:08:58.239733 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:58.239739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:58.239833 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:58.239939 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:58.241501 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:58.242259 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:58.243857 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:08:58.244239 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:08:58.244370 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:58.247833 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:08:58.247837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:08:58.247840 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:08:58.247872 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:08:58.250195 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:58.250992 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:58.250996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:58.250999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:58.251132 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:08:58.251509 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:08:58.251512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:08:58.251514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:08:58.251786 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:08:58.251898 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 22:08:58.252609 (+ 711us) server_negotiation.cc:299] Negotiation successful
0927 22:08:58.252826 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:08:58.254307 3297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:08:58.236785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37251 (local address 127.0.97.3:54839)
0927 22:08:58.237515 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 22:08:58.237551 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:08:58.239456 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:08:58.240165 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:08:58.240174 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:08:58.240776 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:08:58.241341 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:58.241358 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:58.242403 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:58.242414 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:58.243703 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:08:58.243715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:08:58.244452 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:08:58.244460 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:08:58.244702 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:08:58.245414 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:08:58.245436 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:08:58.247624 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:08:58.250338 (+ 2714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:58.250347 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:58.250362 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:58.250864 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:58.251240 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:08:58.251244 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:08:58.251246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:08:58.251415 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:08:58.251901 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:08:58.251908 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:08:58.252246 (+ 338us) client_negotiation.cc:769] Sending connection context
0927 22:08:58.252496 (+ 250us) client_negotiation.cc:240] Negotiation successful
0927 22:08:58.253012 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":311,"threads_started":1}
I20260927 22:08:58.255751 3295 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37251
I20260927 22:08:58.256048 3295 heartbeater.cc:464] Registering TS with master...
I20260927 22:08:58.256752 3295 heartbeater.cc:511] Master 127.0.97.62:37251 requested a full tablet report, sending...
I20260927 22:08:58.258102 2839 ts_manager.cc:194] Registered new tserver with Master: e97ec10b059f4373b1f26dd0197af4d5 (127.0.97.3:46279)
I20260927 22:08:58.259004 2839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54839
I20260927 22:08:58.265817 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:08:58.275621 3028 heartbeater.cc:503] Master 127.0.97.62:37251 was elected leader, sending a full tablet report...
W20260927 22:08:58.296353 3030 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:08:58.293768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56724 (local address 127.0.97.62:37251)
0927 22:08:58.293956 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:08:58.293965 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:08:58.294382 (+ 417us) server_negotiation.cc:407] Connection header received
0927 22:08:58.294635 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:08:58.294645 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:08:58.294748 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:08:58.294861 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:08:58.296095 (+ 1234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56724: BlockingRecv error: recv got EOF from 127.0.0.1:56724 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:08:58.299743 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 2900
W20260927 22:08:58.302672 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 2900: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:38705: server requires authentication, but client does not have Kerberos credentials available
I20260927 22:08:58.302816 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 2900
I20260927 22:08:58.318571 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 3034
W20260927 22:08:58.321240 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 3034: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:34973: server requires authentication, but client does not have Kerberos credentials available
I20260927 22:08:58.321424 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 3034
I20260927 22:08:58.337494 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 3167
W20260927 22:08:58.340348 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 3167: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:46279: server requires authentication, but client does not have Kerberos credentials available
I20260927 22:08:58.340471 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 3167
I20260927 22:08:58.355702 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 2807
I20260927 22:08:58.356891 2869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:08:58.460182 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 2807
2026-09-27T22:08:58Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3491 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:08:59 dist-test-slave-mp03 krb5kdc[3312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:08:59 dist-test-slave-mp03 krb5kdc[3312](info): set up 2 sockets
Sep 27 22:08:59 dist-test-slave-mp03 krb5kdc[3312](info): commencing operation
krb5kdc: starting...
W20260927 22:09:00.570906 388 mini_kdc.cc:121] Time spent starting KDC: real 2.069s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:09:00 dist-test-slave-mp03 krb5kdc[3312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:09:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:09:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:09:00.758315 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45267
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:01.084717 3325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:01.085065 3325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:01.085232 3325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:01.092648 3325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:09:01.092787 3325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:01.092888 3325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:01.092988 3325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:09:01.093060 3325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:09:01.102963 3325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:01.104225 3325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:01.105537 3325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:01.116972 3333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:01.116963 3330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:01.117208 3331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:01.117858 3325 server_base.cc:1034] running on GCE node
I20260927 22:09:01.118692 3325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:01.120144 3325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:01.121351 3325 hybrid_clock.cc:648] HybridClock initialized: now 1790546941121313 us; error 54 us; skew 500 ppm
I20260927 22:09:01.121742 3325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:01 dist-test-slave-mp03 krb5kdc[3312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:01.130264 3325 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:09:01.132076 3325 webserver.cc:460] Webserver started at http://127.0.97.62:33569/ using document root <none> and password file <none>
I20260927 22:09:01.132829 3325 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:01.132974 3325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:01.133281 3325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:01.136444 3325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "61bb88297bf84fc2a64e3bd0fdcb377a"
format_stamp: "Formatted at 2026-09-27 22:09:01 on dist-test-slave-mp03"
server_key: "a5ea469cec63e3d111984410bf139c1c"
server_key_iv: "ad7ce6d6452c37de1522947723934833"
server_key_version: "encryptionkey@0"
I20260927 22:09:01.137243 3325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "61bb88297bf84fc2a64e3bd0fdcb377a"
format_stamp: "Formatted at 2026-09-27 22:09:01 on dist-test-slave-mp03"
server_key: "a5ea469cec63e3d111984410bf139c1c"
server_key_iv: "ad7ce6d6452c37de1522947723934833"
server_key_version: "encryptionkey@0"
I20260927 22:09:01.142494 3325 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:09:01.145947 3340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:01.147326 3325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:09:01.147509 3325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "61bb88297bf84fc2a64e3bd0fdcb377a"
format_stamp: "Formatted at 2026-09-27 22:09:01 on dist-test-slave-mp03"
server_key: "a5ea469cec63e3d111984410bf139c1c"
server_key_iv: "ad7ce6d6452c37de1522947723934833"
server_key_version: "encryptionkey@0"
I20260927 22:09:01.147698 3325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:01.169201 3325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:01.173782 3325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:01.174077 3325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:01.192553 3325 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45267
I20260927 22:09:01.192584 3391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45267 every 8 connection(s)
I20260927 22:09:01.194399 3325 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:09:01.196167 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 3325
I20260927 22:09:01.196360 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:09:01.197043 388 external_mini_cluster.cc:1428] Setting key 8fc06cb6c649c9fb3bb26e3a9539b636
I20260927 22:09:01.198750 3392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:09:01 dist-test-slave-mp03 krb5kdc[3312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:09:01.212004 3392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61bb88297bf84fc2a64e3bd0fdcb377a: Bootstrap starting.
I20260927 22:09:01.216027 3392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61bb88297bf84fc2a64e3bd0fdcb377a: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:01.217269 3392 log.cc:824] T 00000000000000000000000000000000 P 61bb88297bf84fc2a64e3bd0fdcb377a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:01.220047 3392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61bb88297bf84fc2a64e3bd0fdcb377a: No bootstrap required, opened a new log
I20260927 22:09:01.220927 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:01.198946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52928 (local address 127.0.97.62:45267)
0927 22:09:01.199734 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:01.199749 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:01.199796 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:09:01.201820 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:01.201847 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:01.202376 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:01.202847 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:01.204176 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:01.206197 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:01.207511 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:01.208498 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:01.208706 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:01.212814 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:01.212823 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:01.212841 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:01.212887 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:01.216725 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:01.217569 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:01.217576 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:01.217583 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:01.217753 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:01.218136 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:01.218145 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:01.218147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:01.218740 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:01.219043 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:01.219466 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 22:09:01.219860 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":335,"threads_started":1}
I20260927 22:09:01.224911 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:09:01.227983 3392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61bb88297bf84fc2a64e3bd0fdcb377a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61bb88297bf84fc2a64e3bd0fdcb377a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45267 } }
I20260927 22:09:01.228320 3392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61bb88297bf84fc2a64e3bd0fdcb377a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:01.228421 3392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61bb88297bf84fc2a64e3bd0fdcb377a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61bb88297bf84fc2a64e3bd0fdcb377a, State: Initialized, Role: FOLLOWER
I20260927 22:09:01.229018 3392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61bb88297bf84fc2a64e3bd0fdcb377a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61bb88297bf84fc2a64e3bd0fdcb377a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45267 } }
I20260927 22:09:01.229242 3392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61bb88297bf84fc2a64e3bd0fdcb377a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:09:01.229368 3392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61bb88297bf84fc2a64e3bd0fdcb377a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:09:01.229538 3392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61bb88297bf84fc2a64e3bd0fdcb377a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:01.231923 3392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61bb88297bf84fc2a64e3bd0fdcb377a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61bb88297bf84fc2a64e3bd0fdcb377a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45267 } }
I20260927 22:09:01.232468 3392 leader_election.cc:302] T 00000000000000000000000000000000 P 61bb88297bf84fc2a64e3bd0fdcb377a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61bb88297bf84fc2a64e3bd0fdcb377a; no voters:
I20260927 22:09:01.233093 3392 leader_election.cc:288] T 00000000000000000000000000000000 P 61bb88297bf84fc2a64e3bd0fdcb377a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:09:01.233213 3398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61bb88297bf84fc2a64e3bd0fdcb377a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:09:01.234473 3398 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61bb88297bf84fc2a64e3bd0fdcb377a [term 1 LEADER]: Becoming Leader. State: Replica: 61bb88297bf84fc2a64e3bd0fdcb377a, State: Running, Role: LEADER
I20260927 22:09:01.235029 3398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61bb88297bf84fc2a64e3bd0fdcb377a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61bb88297bf84fc2a64e3bd0fdcb377a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45267 } }
I20260927 22:09:01.235354 3392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61bb88297bf84fc2a64e3bd0fdcb377a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:09:01.237730 3400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61bb88297bf84fc2a64e3bd0fdcb377a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61bb88297bf84fc2a64e3bd0fdcb377a. Latest consensus state: current_term: 1 leader_uuid: "61bb88297bf84fc2a64e3bd0fdcb377a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61bb88297bf84fc2a64e3bd0fdcb377a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45267 } } }
I20260927 22:09:01.238091 3400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61bb88297bf84fc2a64e3bd0fdcb377a [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:01.237819 3399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61bb88297bf84fc2a64e3bd0fdcb377a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61bb88297bf84fc2a64e3bd0fdcb377a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61bb88297bf84fc2a64e3bd0fdcb377a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45267 } } }
I20260927 22:09:01.238232 3399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61bb88297bf84fc2a64e3bd0fdcb377a [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:01.249121 3404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:09:01.254485 3404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:09:01.269098 3404 catalog_manager.cc:1347] Generated new cluster ID: 5d11c70e5a82469bb73b596ba2b0606a
I20260927 22:09:01.269235 3404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:09:01.280412 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:01.271454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52934 (local address 127.0.97.62:45267)
0927 22:09:01.271603 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:01.271610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:01.272141 (+ 531us) server_negotiation.cc:407] Connection header received
0927 22:09:01.272305 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:01.272311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:01.272393 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:01.272503 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:01.273623 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:01.274867 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:01.276575 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:01.277246 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:01.277372 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:01.278541 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:01.278548 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:01.278553 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:09:01.279516 (+ 963us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:01.279575 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:09:01.279610 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:01.279722 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:01.279880 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 22:09:01.279978 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:09:01.282054 3404 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:09:01.283455 3404 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:09:01.292433 3404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61bb88297bf84fc2a64e3bd0fdcb377a: Generated new TSK 0
I20260927 22:09:01.293267 3404 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:09:01.308331 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 3325
I20260927 22:09:01.309443 3387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:01.398219 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 3325
2026-09-27T22:09:01Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2932 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:09:01 dist-test-slave-mp03 krb5kdc[3426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:09:01 dist-test-slave-mp03 krb5kdc[3426](info): set up 2 sockets
Sep 27 22:09:01 dist-test-slave-mp03 krb5kdc[3426](info): commencing operation
krb5kdc: starting...
W20260927 22:09:02.483003 388 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:09:02 dist-test-slave-mp03 krb5kdc[3426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:09:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:09:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:09:02.670153 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45813
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:02.993320 3439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:02.993716 3439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:02.993875 3439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:03.001299 3439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:09:03.001461 3439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:03.001570 3439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:03.001655 3439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:09:03.001719 3439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:09:03.011647 3439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:03.013069 3439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:03.014403 3439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:03.026180 3447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:03.026271 3445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:03.026443 3444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:03.027576 3439 server_base.cc:1034] running on GCE node
I20260927 22:09:03.028316 3439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:03.029707 3439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:03.030910 3439 hybrid_clock.cc:648] HybridClock initialized: now 1790546943030871 us; error 31 us; skew 500 ppm
I20260927 22:09:03.031275 3439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:03 dist-test-slave-mp03 krb5kdc[3426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:03.034376 3439 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:09:03.035895 3439 webserver.cc:460] Webserver started at http://127.0.97.62:42977/ using document root <none> and password file <none>
I20260927 22:09:03.036566 3439 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:03.036707 3439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:03.037060 3439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:03.040089 3439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "b832eb05373a44d1af08324283d94f3a"
format_stamp: "Formatted at 2026-09-27 22:09:03 on dist-test-slave-mp03"
server_key: "ed976c8ca136f45892aef22a9a1c3090"
server_key_iv: "ae2a1710870b163bc552d3281c7a8b19"
server_key_version: "encryptionkey@0"
I20260927 22:09:03.040866 3439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "b832eb05373a44d1af08324283d94f3a"
format_stamp: "Formatted at 2026-09-27 22:09:03 on dist-test-slave-mp03"
server_key: "ed976c8ca136f45892aef22a9a1c3090"
server_key_iv: "ae2a1710870b163bc552d3281c7a8b19"
server_key_version: "encryptionkey@0"
I20260927 22:09:03.046798 3439 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 22:09:03.052079 3454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:03.053903 3439 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:09:03.054085 3439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "b832eb05373a44d1af08324283d94f3a"
format_stamp: "Formatted at 2026-09-27 22:09:03 on dist-test-slave-mp03"
server_key: "ed976c8ca136f45892aef22a9a1c3090"
server_key_iv: "ae2a1710870b163bc552d3281c7a8b19"
server_key_version: "encryptionkey@0"
I20260927 22:09:03.054303 3439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:03.078677 3439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:03.084347 3439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:03.084694 3439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:03.105134 3439 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45813
I20260927 22:09:03.105152 3505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45813 every 8 connection(s)
I20260927 22:09:03.106484 3439 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:09:03.108193 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 3439
I20260927 22:09:03.108428 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:09:03.109082 388 external_mini_cluster.cc:1428] Setting key c7bd46a68b1cde72b884d800b0361aba
I20260927 22:09:03.110556 3506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:09:03 dist-test-slave-mp03 krb5kdc[3426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:09:03.124231 3506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b832eb05373a44d1af08324283d94f3a: Bootstrap starting.
I20260927 22:09:03.128454 3506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b832eb05373a44d1af08324283d94f3a: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:03.129658 3506 log.cc:824] T 00000000000000000000000000000000 P b832eb05373a44d1af08324283d94f3a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:03.133179 3506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b832eb05373a44d1af08324283d94f3a: No bootstrap required, opened a new log
I20260927 22:09:03.133285 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:03.111193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36478 (local address 127.0.97.62:45813)
0927 22:09:03.111955 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:03.111972 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:03.112019 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:09:03.113798 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:03.113833 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:03.114340 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:03.114810 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:03.115985 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:03.118010 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:03.119270 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:03.120278 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:03.120475 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:03.124702 (+ 4227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:03.124713 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:03.124756 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:03.124804 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:03.128657 (+ 3853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:03.129441 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:03.129453 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:03.129469 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:03.129685 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:03.130119 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:03.130127 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:03.130133 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:03.130845 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:03.131108 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:03.131534 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 22:09:03.131958 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":319,"threads_started":1}
I20260927 22:09:03.141203 3506 raft_consensus.cc:348] T 00000000000000000000000000000000 P b832eb05373a44d1af08324283d94f3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b832eb05373a44d1af08324283d94f3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45813 } }
I20260927 22:09:03.141556 3506 raft_consensus.cc:374] T 00000000000000000000000000000000 P b832eb05373a44d1af08324283d94f3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:03.141652 3506 raft_consensus.cc:729] T 00000000000000000000000000000000 P b832eb05373a44d1af08324283d94f3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b832eb05373a44d1af08324283d94f3a, State: Initialized, Role: FOLLOWER
I20260927 22:09:03.142153 3506 consensus_queue.cc:260] T 00000000000000000000000000000000 P b832eb05373a44d1af08324283d94f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b832eb05373a44d1af08324283d94f3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45813 } }
I20260927 22:09:03.142352 3506 raft_consensus.cc:388] T 00000000000000000000000000000000 P b832eb05373a44d1af08324283d94f3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:09:03.142450 3506 raft_consensus.cc:482] T 00000000000000000000000000000000 P b832eb05373a44d1af08324283d94f3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:09:03.142565 3506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b832eb05373a44d1af08324283d94f3a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:03.144917 3506 raft_consensus.cc:504] T 00000000000000000000000000000000 P b832eb05373a44d1af08324283d94f3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b832eb05373a44d1af08324283d94f3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45813 } }
I20260927 22:09:03.145357 3506 leader_election.cc:302] T 00000000000000000000000000000000 P b832eb05373a44d1af08324283d94f3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b832eb05373a44d1af08324283d94f3a; no voters:
I20260927 22:09:03.145975 3506 leader_election.cc:288] T 00000000000000000000000000000000 P b832eb05373a44d1af08324283d94f3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:09:03.146194 3511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b832eb05373a44d1af08324283d94f3a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:09:03.147078 3511 raft_consensus.cc:686] T 00000000000000000000000000000000 P b832eb05373a44d1af08324283d94f3a [term 1 LEADER]: Becoming Leader. State: Replica: b832eb05373a44d1af08324283d94f3a, State: Running, Role: LEADER
I20260927 22:09:03.147691 3511 consensus_queue.cc:237] T 00000000000000000000000000000000 P b832eb05373a44d1af08324283d94f3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b832eb05373a44d1af08324283d94f3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45813 } }
I20260927 22:09:03.148067 3506 sys_catalog.cc:564] T 00000000000000000000000000000000 P b832eb05373a44d1af08324283d94f3a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:09:03.151321 3512 sys_catalog.cc:455] T 00000000000000000000000000000000 P b832eb05373a44d1af08324283d94f3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b832eb05373a44d1af08324283d94f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b832eb05373a44d1af08324283d94f3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45813 } } }
I20260927 22:09:03.151818 3512 sys_catalog.cc:458] T 00000000000000000000000000000000 P b832eb05373a44d1af08324283d94f3a [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:03.151295 3513 sys_catalog.cc:455] T 00000000000000000000000000000000 P b832eb05373a44d1af08324283d94f3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b832eb05373a44d1af08324283d94f3a. Latest consensus state: current_term: 1 leader_uuid: "b832eb05373a44d1af08324283d94f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b832eb05373a44d1af08324283d94f3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45813 } } }
I20260927 22:09:03.152662 3513 sys_catalog.cc:458] T 00000000000000000000000000000000 P b832eb05373a44d1af08324283d94f3a [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:03.154047 3519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:09:03.160492 3519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:09:03.171206 3519 catalog_manager.cc:1347] Generated new cluster ID: 61128b95ecd5444b9063166e460311f7
I20260927 22:09:03.171334 3519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:09:03.193975 3519 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:09:03.195380 3519 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:09:03.216027 3519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b832eb05373a44d1af08324283d94f3a: Generated new TSK 0
I20260927 22:09:03.216972 3519 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:09:03.279848 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45813
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:03.628633 3532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:03.629009 3532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:03.629189 3532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:03.637074 3532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:03.637172 3532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:03.637360 3532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:09:03.647102 3532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:03.648488 3532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:03.649948 3532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:03.663089 3537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:03.663149 3538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:03.663149 3540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:03.663868 3532 server_base.cc:1034] running on GCE node
I20260927 22:09:03.664438 3532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:03.665378 3532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:03.666580 3532 hybrid_clock.cc:648] HybridClock initialized: now 1790546943666526 us; error 49 us; skew 500 ppm
I20260927 22:09:03.666972 3532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:03 dist-test-slave-mp03 krb5kdc[3426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:03.670015 3532 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:09:03.671497 3532 webserver.cc:460] Webserver started at http://127.0.97.1:34333/ using document root <none> and password file <none>
I20260927 22:09:03.672137 3532 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:03.672235 3532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:03.672483 3532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:03.675386 3532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "10cb43e972b34eff8ae9c5f8290e8a74"
format_stamp: "Formatted at 2026-09-27 22:09:03 on dist-test-slave-mp03"
server_key: "c0a29eaf57b700c38ff1cdf052330972"
server_key_iv: "cb95af7a51669b2afe66c8be4fff3ec2"
server_key_version: "encryptionkey@0"
I20260927 22:09:03.676105 3532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "10cb43e972b34eff8ae9c5f8290e8a74"
format_stamp: "Formatted at 2026-09-27 22:09:03 on dist-test-slave-mp03"
server_key: "c0a29eaf57b700c38ff1cdf052330972"
server_key_iv: "cb95af7a51669b2afe66c8be4fff3ec2"
server_key_version: "encryptionkey@0"
I20260927 22:09:03.681584 3532 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:09:03.684844 3547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:03.686131 3532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:09:03.686291 3532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "10cb43e972b34eff8ae9c5f8290e8a74"
format_stamp: "Formatted at 2026-09-27 22:09:03 on dist-test-slave-mp03"
server_key: "c0a29eaf57b700c38ff1cdf052330972"
server_key_iv: "cb95af7a51669b2afe66c8be4fff3ec2"
server_key_version: "encryptionkey@0"
I20260927 22:09:03.686482 3532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:03.710632 3532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:03.715011 3532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:03.715317 3532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:03.716289 3532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:03.717566 3532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:03.717667 3532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:03.717808 3532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:03.717880 3532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:03.750321 3532 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33301
I20260927 22:09:03.750409 3659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33301 every 8 connection(s)
I20260927 22:09:03.751690 3532 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:09:03.761457 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 3532
I20260927 22:09:03.761704 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:09:03.762429 388 external_mini_cluster.cc:1428] Setting key ea88b4857d9d2ae9a5dbe7da78192358
Sep 27 22:09:03 dist-test-slave-mp03 krb5kdc[3426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:09:03.772970 3662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:03.754175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59019 (local address 127.0.97.62:45813)
0927 22:09:03.754588 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:03.754632 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:03.755667 (+ 1035us) server_negotiation.cc:407] Connection header received
0927 22:09:03.757438 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:03.757444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:03.757529 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:03.757642 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:03.759175 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:03.759867 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:03.761586 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:03.762031 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:03.762124 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:03.766225 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:03.766235 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:03.766242 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:03.766287 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:03.769795 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:03.770745 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:03.770749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:03.770751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:03.770878 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:03.771360 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:03.771363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:03.771364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:03.771616 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:03.771718 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:03.772573 (+ 855us) server_negotiation.cc:299] Negotiation successful
0927 22:09:03.772725 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":198,"threads_started":1}
I20260927 22:09:03.774526 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:03.754852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45813 (local address 127.0.97.1:59019)
0927 22:09:03.755516 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:03.755553 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:03.757181 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:03.757863 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:03.757871 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:03.758401 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:03.759003 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:03.759021 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:03.760017 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:03.760021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:03.761415 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:03.761431 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:03.762252 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:03.762261 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:03.762559 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:03.763312 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:03.763350 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:03.765999 (+ 2649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:03.769936 (+ 3937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:03.769943 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:03.769956 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:03.770612 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:03.770989 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:03.770996 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:03.771001 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:03.771252 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:03.771740 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:03.771750 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:03.772185 (+ 435us) client_negotiation.cc:769] Sending connection context
0927 22:09:03.772486 (+ 301us) client_negotiation.cc:240] Negotiation successful
0927 22:09:03.773055 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":263,"threads_started":1}
I20260927 22:09:03.776027 3660 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45813
I20260927 22:09:03.776327 3660 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:03.777040 3660 heartbeater.cc:511] Master 127.0.97.62:45813 requested a full tablet report, sending...
I20260927 22:09:03.779146 3471 ts_manager.cc:194] Registered new tserver with Master: 10cb43e972b34eff8ae9c5f8290e8a74 (127.0.97.1:33301)
I20260927 22:09:03.780555 3471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59019
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:09:03.806401 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45813
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:04.127535 3666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:04.127805 3666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:04.127995 3666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:04.135303 3666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:04.135437 3666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:04.135624 3666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:09:04.145251 3666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:04.146674 3666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:04.148073 3666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:04.161417 3671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:04.161383 3672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:04.161548 3674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:04.162184 3666 server_base.cc:1034] running on GCE node
I20260927 22:09:04.162930 3666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:04.164187 3666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:04.165465 3666 hybrid_clock.cc:648] HybridClock initialized: now 1790546944165396 us; error 53 us; skew 500 ppm
I20260927 22:09:04.165836 3666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:04 dist-test-slave-mp03 krb5kdc[3426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:04.169305 3666 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:09:04.170790 3666 webserver.cc:460] Webserver started at http://127.0.97.2:39527/ using document root <none> and password file <none>
I20260927 22:09:04.171461 3666 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:04.171566 3666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:04.171846 3666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:04.174966 3666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "0b98ff54ad8449a5adc814713a1197d7"
format_stamp: "Formatted at 2026-09-27 22:09:04 on dist-test-slave-mp03"
server_key: "9ae7a7c67196babbce41a438661d1194"
server_key_iv: "1c59dbcee458a06def66a53dae36da16"
server_key_version: "encryptionkey@0"
I20260927 22:09:04.175750 3666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0b98ff54ad8449a5adc814713a1197d7"
format_stamp: "Formatted at 2026-09-27 22:09:04 on dist-test-slave-mp03"
server_key: "9ae7a7c67196babbce41a438661d1194"
server_key_iv: "1c59dbcee458a06def66a53dae36da16"
server_key_version: "encryptionkey@0"
I20260927 22:09:04.181068 3666 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:09:04.184742 3681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:04.186142 3666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:09:04.186285 3666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "0b98ff54ad8449a5adc814713a1197d7"
format_stamp: "Formatted at 2026-09-27 22:09:04 on dist-test-slave-mp03"
server_key: "9ae7a7c67196babbce41a438661d1194"
server_key_iv: "1c59dbcee458a06def66a53dae36da16"
server_key_version: "encryptionkey@0"
I20260927 22:09:04.186514 3666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:04.217679 3666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:04.221776 3666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:04.222064 3666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:04.223145 3666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:04.224699 3666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:04.224848 3666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:04.224982 3666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:04.225059 3666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:04.257023 3666 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42259
I20260927 22:09:04.257071 3793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42259 every 8 connection(s)
I20260927 22:09:04.258484 3666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:09:04.264954 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 3666
I20260927 22:09:04.265121 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:09:04.265558 388 external_mini_cluster.cc:1428] Setting key b0cd8dec5bbc9091e46b8e124c373bbe
Sep 27 22:09:04 dist-test-slave-mp03 krb5kdc[3426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:09:04.277631 3662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:04.260989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55947 (local address 127.0.97.62:45813)
0927 22:09:04.261162 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:04.261166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:04.262271 (+ 1105us) server_negotiation.cc:407] Connection header received
0927 22:09:04.264039 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:04.264044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:04.264109 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:04.264194 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:04.265781 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:04.266654 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:04.268177 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:04.268595 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:04.268722 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:04.272048 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:04.272054 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:04.272058 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:04.272099 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:04.274727 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:04.275397 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:04.275404 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:04.275407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:04.275541 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:04.275938 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:04.275945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:04.275948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:04.276204 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:04.276348 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:04.277066 (+ 718us) server_negotiation.cc:299] Negotiation successful
0927 22:09:04.277279 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:09:04.278766 3796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:04.261420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45813 (local address 127.0.97.2:55947)
0927 22:09:04.262101 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:04.262143 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:04.263800 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:04.264414 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:04.264421 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:04.264903 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:04.265578 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:04.265616 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:04.266820 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:04.266825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:04.268040 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:04.268051 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:04.268808 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:04.268815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:04.269042 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:04.269729 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:04.269751 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:04.271849 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:04.274855 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:04.274866 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:04.274883 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:04.275281 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:04.275647 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:04.275652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:04.275656 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:04.275833 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:04.276324 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:04.276332 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:04.276720 (+ 388us) client_negotiation.cc:769] Sending connection context
0927 22:09:04.276995 (+ 275us) client_negotiation.cc:240] Negotiation successful
0927 22:09:04.277519 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":273,"threads_started":1}
I20260927 22:09:04.280395 3794 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45813
I20260927 22:09:04.280719 3794 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:04.281459 3794 heartbeater.cc:511] Master 127.0.97.62:45813 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:09:04.283015 3471 ts_manager.cc:194] Registered new tserver with Master: 0b98ff54ad8449a5adc814713a1197d7 (127.0.97.2:42259)
I20260927 22:09:04.283912 3471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:09:04.307833 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45813
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:04.625510 3799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:04.625837 3799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:04.626010 3799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:04.633378 3799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:04.633546 3799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:04.633738 3799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:09:04.643402 3799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:04.644721 3799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:04.646006 3799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:04.658948 3807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:04.659147 3804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:04.658977 3805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:04.659725 3799 server_base.cc:1034] running on GCE node
I20260927 22:09:04.660310 3799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:04.661204 3799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:04.662392 3799 hybrid_clock.cc:648] HybridClock initialized: now 1790546944662336 us; error 49 us; skew 500 ppm
I20260927 22:09:04.662745 3799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:04 dist-test-slave-mp03 krb5kdc[3426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:04.665773 3799 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:09:04.667277 3799 webserver.cc:460] Webserver started at http://127.0.97.3:39379/ using document root <none> and password file <none>
I20260927 22:09:04.667984 3799 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:04.668108 3799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:04.668437 3799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:04.671291 3799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "84316513cc6b4c27808277c6c8535156"
format_stamp: "Formatted at 2026-09-27 22:09:04 on dist-test-slave-mp03"
server_key: "5c451e90f57db5a4dcfff746c35dfecd"
server_key_iv: "09cef6600dfd79a6c06236f06c9a2262"
server_key_version: "encryptionkey@0"
I20260927 22:09:04.672037 3799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "84316513cc6b4c27808277c6c8535156"
format_stamp: "Formatted at 2026-09-27 22:09:04 on dist-test-slave-mp03"
server_key: "5c451e90f57db5a4dcfff746c35dfecd"
server_key_iv: "09cef6600dfd79a6c06236f06c9a2262"
server_key_version: "encryptionkey@0"
I20260927 22:09:04.676879 3799 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:09:04.680256 3814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:04.681514 3799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:09:04.681675 3799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "84316513cc6b4c27808277c6c8535156"
format_stamp: "Formatted at 2026-09-27 22:09:04 on dist-test-slave-mp03"
server_key: "5c451e90f57db5a4dcfff746c35dfecd"
server_key_iv: "09cef6600dfd79a6c06236f06c9a2262"
server_key_version: "encryptionkey@0"
I20260927 22:09:04.681871 3799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:04.695058 3799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:04.699258 3799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:04.699553 3799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:04.700482 3799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:04.701783 3799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:04.701854 3799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:04.701982 3799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:04.702031 3799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:04.734386 3799 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42325
I20260927 22:09:04.734428 3926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42325 every 8 connection(s)
I20260927 22:09:04.735737 3799 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:09:04.745674 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 3799
I20260927 22:09:04.745893 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:09:04.746327 388 external_mini_cluster.cc:1428] Setting key 766f34badf579f8ef6d5dd6ce977d4e7
Sep 27 22:09:04 dist-test-slave-mp03 krb5kdc[3426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:09:04.755671 3662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:04.738238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45619 (local address 127.0.97.62:45813)
0927 22:09:04.738415 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:04.738419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:04.739573 (+ 1154us) server_negotiation.cc:407] Connection header received
0927 22:09:04.741355 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:04.741361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:04.741450 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:04.741558 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:04.743032 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:04.743752 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:04.745250 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:04.745655 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:04.745784 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:04.749645 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:04.749652 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:04.749656 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:04.749708 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:04.752313 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:04.753207 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:04.753217 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:04.753219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:04.753344 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:04.753921 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:04.753925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:04.753927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:04.754207 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:04.754315 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:04.755272 (+ 957us) server_negotiation.cc:299] Negotiation successful
0927 22:09:04.755432 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:09:04.756644 3929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:04.738744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45813 (local address 127.0.97.3:45619)
0927 22:09:04.739417 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:04.739453 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:04.741101 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:04.741784 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:04.741793 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:04.742264 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:04.742867 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:04.742885 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:04.743891 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:04.743898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:04.745114 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:04.745125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:04.745888 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:04.745895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:04.746147 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:04.746841 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:04.746864 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:04.749427 (+ 2563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:04.752460 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:04.752469 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:04.752484 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:04.753080 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:04.753552 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:04.753558 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:04.753562 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:04.753798 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:04.754337 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:04.754345 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:04.754848 (+ 503us) client_negotiation.cc:769] Sending connection context
0927 22:09:04.755161 (+ 313us) client_negotiation.cc:240] Negotiation successful
0927 22:09:04.755546 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":263,"threads_started":1}
I20260927 22:09:04.758044 3927 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45813
I20260927 22:09:04.758347 3927 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:04.759219 3927 heartbeater.cc:511] Master 127.0.97.62:45813 requested a full tablet report, sending...
I20260927 22:09:04.761252 3470 ts_manager.cc:194] Registered new tserver with Master: 84316513cc6b4c27808277c6c8535156 (127.0.97.3:42325)
I20260927 22:09:04.762276 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:09:04.762471 3470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45619
I20260927 22:09:04.783445 3660 heartbeater.cc:503] Master 127.0.97.62:45813 was elected leader, sending a full tablet report...
Sep 27 22:09:04 dist-test-slave-mp03 krb5kdc[3426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546944, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 22:09:04 dist-test-slave-mp03 krb5kdc[3426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546944, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:09:04.810398 3662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:04.796761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36490 (local address 127.0.97.62:45813)
0927 22:09:04.796979 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:04.796989 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:04.797369 (+ 380us) server_negotiation.cc:407] Connection header received
0927 22:09:04.797510 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:04.797514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:04.797586 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:04.797672 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:04.799079 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:04.799813 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:04.800985 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:04.801345 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:04.801436 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:04.805624 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:04.805629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:04.805632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:04.805667 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:04.808130 (+ 2463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:04.808735 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:04.808738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:04.808741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:04.808850 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:04.809179 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:04.809182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:04.809184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:04.809484 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:04.809672 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:04.809956 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 22:09:04.810127 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 22:09:04.811206 3470 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36490
I20260927 22:09:04.812592 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 3532
Sep 27 22:09:04 dist-test-slave-mp03 krb5kdc[3426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546944, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:09:04.838557 3938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:04.813591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42602 (local address 127.0.97.1:33301)
0927 22:09:04.814492 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:04.814501 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:04.823180 (+ 8679us) server_negotiation.cc:407] Connection header received
0927 22:09:04.823407 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:04.823415 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:04.823777 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:04.823963 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:04.825564 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:04.826748 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:04.828011 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:04.828593 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:04.828699 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:04.832582 (+ 3883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:04.832593 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:04.832607 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:04.832640 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:04.835557 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:04.836194 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:04.836201 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:04.836203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:04.836505 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:04.837107 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:04.837110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:04.837112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:04.837476 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:04.837706 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:04.837990 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 22:09:04.838245 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":566,"threads_started":1}
W20260927 22:09:04.839903 3655 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42602
W20260927 22:09:04.841082 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 3532: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:09:04.841311 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 3532
I20260927 22:09:04.857419 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 3666
Sep 27 22:09:04 dist-test-slave-mp03 krb5kdc[3426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546944, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:09:04.872790 3939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:04.858336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55956 (local address 127.0.97.2:42259)
0927 22:09:04.858875 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:04.858883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:04.858909 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:09:04.858968 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:04.858973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:04.859241 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:04.859371 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:04.860570 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:04.861515 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:04.862796 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:04.863312 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:04.863413 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:04.867426 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:04.867437 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:04.867453 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:04.867485 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:04.870229 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:04.870940 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:04.870947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:04.870950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:04.871076 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:04.871386 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:04.871389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:04.871391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:04.871730 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:04.871928 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:04.872189 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 22:09:04.872408 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":289,"threads_started":1}
W20260927 22:09:04.874054 3789 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55956
W20260927 22:09:04.875198 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 3666: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:09:04.875327 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 3666
I20260927 22:09:04.890420 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 3799
Sep 27 22:09:04 dist-test-slave-mp03 krb5kdc[3426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546944, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:09:04.906574 3940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:04.891510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52412 (local address 127.0.97.3:42325)
0927 22:09:04.892176 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:04.892184 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:04.892211 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:09:04.892268 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:04.892274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:04.892482 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:04.892615 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:04.893862 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:04.894966 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:04.896347 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:04.896886 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:04.896978 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:04.901310 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:04.901323 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:04.901338 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:04.901373 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:04.904315 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:04.904919 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:04.904926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:04.904928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:04.905056 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:04.905349 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:04.905352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:04.905353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:04.905676 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:04.905901 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:04.906106 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 22:09:04.906295 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":342,"threads_started":1}
W20260927 22:09:04.907936 3922 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52412
W20260927 22:09:04.909024 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 3799: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:09:04.909166 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 3799
I20260927 22:09:04.924584 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 3439
I20260927 22:09:04.925701 3501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:05.045733 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 3439
2026-09-27T22:09:05Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3654 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:09:05 dist-test-slave-mp03 krb5kdc[3944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:09:05 dist-test-slave-mp03 krb5kdc[3944](info): set up 2 sockets
Sep 27 22:09:05 dist-test-slave-mp03 krb5kdc[3944](info): commencing operation
krb5kdc: starting...
W20260927 22:09:06.131079 388 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:09:06 dist-test-slave-mp03 krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:09:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:09:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:09:06.322781 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39933
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:06.642527 3957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:06.642915 3957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:06.643081 3957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:06.650373 3957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:09:06.650478 3957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:06.650579 3957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:06.650705 3957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:09:06.650777 3957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:09:06.660791 3957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:06.662127 3957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:06.663496 3957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:06.675057 3962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:06.674989 3965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:06.675030 3963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:06.676040 3957 server_base.cc:1034] running on GCE node
I20260927 22:09:06.676770 3957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:06.678083 3957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:06.679275 3957 hybrid_clock.cc:648] HybridClock initialized: now 1790546946679212 us; error 56 us; skew 500 ppm
I20260927 22:09:06.679620 3957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:06 dist-test-slave-mp03 krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:06.689023 3957 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:09:06.690639 3957 webserver.cc:460] Webserver started at http://127.0.97.62:44455/ using document root <none> and password file <none>
I20260927 22:09:06.691325 3957 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:06.691423 3957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:06.691681 3957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:06.694643 3957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "50fa940e990941068806e66ee45ef00e"
format_stamp: "Formatted at 2026-09-27 22:09:06 on dist-test-slave-mp03"
server_key: "b4fab1028d17faf319423107f40024a3"
server_key_iv: "54355b4815203cd7f783673b77beada2"
server_key_version: "encryptionkey@0"
I20260927 22:09:06.695386 3957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "50fa940e990941068806e66ee45ef00e"
format_stamp: "Formatted at 2026-09-27 22:09:06 on dist-test-slave-mp03"
server_key: "b4fab1028d17faf319423107f40024a3"
server_key_iv: "54355b4815203cd7f783673b77beada2"
server_key_version: "encryptionkey@0"
I20260927 22:09:06.700459 3957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:09:06.703755 3972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:06.705098 3957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 22:09:06.705273 3957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "50fa940e990941068806e66ee45ef00e"
format_stamp: "Formatted at 2026-09-27 22:09:06 on dist-test-slave-mp03"
server_key: "b4fab1028d17faf319423107f40024a3"
server_key_iv: "54355b4815203cd7f783673b77beada2"
server_key_version: "encryptionkey@0"
I20260927 22:09:06.705463 3957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:06.720247 3957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:06.724151 3957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:06.724574 3957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:06.743307 3957 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39933
I20260927 22:09:06.743323 4023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39933 every 8 connection(s)
I20260927 22:09:06.744792 3957 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:09:06.748513 4024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:06.750777 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 3957
I20260927 22:09:06.751000 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:09:06.751754 388 external_mini_cluster.cc:1428] Setting key 9ed09b28a73dd0d933681b2dde2a0e89
I20260927 22:09:06.761344 4024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50fa940e990941068806e66ee45ef00e: Bootstrap starting.
I20260927 22:09:06.764514 4024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50fa940e990941068806e66ee45ef00e: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:06.765465 4024 log.cc:824] T 00000000000000000000000000000000 P 50fa940e990941068806e66ee45ef00e: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:09:06 dist-test-slave-mp03 krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:09:06.768232 4024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50fa940e990941068806e66ee45ef00e: No bootstrap required, opened a new log
I20260927 22:09:06.774658 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:06.754196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55108 (local address 127.0.97.62:39933)
0927 22:09:06.755520 (+ 1324us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:06.755532 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:06.755573 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:09:06.757264 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:06.757337 (+ 73us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:06.757854 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:06.758200 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:06.760049 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:06.761376 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:06.762762 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:06.763426 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:06.763612 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:06.767466 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:06.767474 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:06.767488 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:06.767525 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:06.770651 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:06.771335 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:06.771341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:06.771347 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:06.771507 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:06.771888 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:06.771900 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:06.771902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:06.772420 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:06.772767 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:06.773175 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 22:09:06.773577 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1048,"thread_start_us":792,"threads_started":1}
I20260927 22:09:06.776365 4024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50fa940e990941068806e66ee45ef00e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50fa940e990941068806e66ee45ef00e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39933 } }
I20260927 22:09:06.776798 4024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50fa940e990941068806e66ee45ef00e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:06.776909 4024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50fa940e990941068806e66ee45ef00e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fa940e990941068806e66ee45ef00e, State: Initialized, Role: FOLLOWER
I20260927 22:09:06.777513 4024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50fa940e990941068806e66ee45ef00e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50fa940e990941068806e66ee45ef00e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39933 } }
I20260927 22:09:06.777810 4024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50fa940e990941068806e66ee45ef00e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:09:06.777940 4024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50fa940e990941068806e66ee45ef00e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:09:06.778053 4024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50fa940e990941068806e66ee45ef00e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:06.780112 4024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50fa940e990941068806e66ee45ef00e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50fa940e990941068806e66ee45ef00e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39933 } }
I20260927 22:09:06.780522 4024 leader_election.cc:302] T 00000000000000000000000000000000 P 50fa940e990941068806e66ee45ef00e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50fa940e990941068806e66ee45ef00e; no voters:
I20260927 22:09:06.781002 4024 leader_election.cc:288] T 00000000000000000000000000000000 P 50fa940e990941068806e66ee45ef00e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:09:06.781152 4029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50fa940e990941068806e66ee45ef00e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:09:06.781811 4029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50fa940e990941068806e66ee45ef00e [term 1 LEADER]: Becoming Leader. State: Replica: 50fa940e990941068806e66ee45ef00e, State: Running, Role: LEADER
I20260927 22:09:06.782299 4029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50fa940e990941068806e66ee45ef00e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50fa940e990941068806e66ee45ef00e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39933 } }
I20260927 22:09:06.783070 4024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50fa940e990941068806e66ee45ef00e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:09:06.785252 4031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50fa940e990941068806e66ee45ef00e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50fa940e990941068806e66ee45ef00e. Latest consensus state: current_term: 1 leader_uuid: "50fa940e990941068806e66ee45ef00e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50fa940e990941068806e66ee45ef00e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39933 } } }
I20260927 22:09:06.785224 4030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50fa940e990941068806e66ee45ef00e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50fa940e990941068806e66ee45ef00e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50fa940e990941068806e66ee45ef00e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39933 } } }
I20260927 22:09:06.785785 4030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50fa940e990941068806e66ee45ef00e [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:06.785786 4031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50fa940e990941068806e66ee45ef00e [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:06.786666 4038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:09:06.791955 4038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:09:06.806548 4038 catalog_manager.cc:1347] Generated new cluster ID: 434d3129a93d4d048e59ca4129702096
I20260927 22:09:06.806708 4038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:09:06.832775 4038 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:09:06.834143 4038 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:09:06.839722 4038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50fa940e990941068806e66ee45ef00e: Generated new TSK 0
I20260927 22:09:06.840788 4038 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:09:06.886123 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39933
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:07.217221 4050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:07.217494 4050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:07.217661 4050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:07.224983 4050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:07.225085 4050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:07.225281 4050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:09:07.235322 4050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:07.236608 4050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:07.237922 4050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:07.250703 4055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:07.250577 4058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:07.250742 4056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:07.251868 4050 server_base.cc:1034] running on GCE node
I20260927 22:09:07.252439 4050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:07.253355 4050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:07.254576 4050 hybrid_clock.cc:648] HybridClock initialized: now 1790546947254529 us; error 42 us; skew 500 ppm
I20260927 22:09:07.254932 4050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:07 dist-test-slave-mp03 krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:07.257992 4050 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:09:07.259491 4050 webserver.cc:460] Webserver started at http://127.0.97.1:39071/ using document root <none> and password file <none>
I20260927 22:09:07.260149 4050 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:07.260290 4050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:07.260593 4050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:07.263446 4050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "33ae3061c0b54a80a4275b6999e5bda4"
format_stamp: "Formatted at 2026-09-27 22:09:07 on dist-test-slave-mp03"
server_key: "473e4655bc24942a726ed4efb0b3d67e"
server_key_iv: "da5cde3408efc710907f921e92301cc9"
server_key_version: "encryptionkey@0"
I20260927 22:09:07.264194 4050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "33ae3061c0b54a80a4275b6999e5bda4"
format_stamp: "Formatted at 2026-09-27 22:09:07 on dist-test-slave-mp03"
server_key: "473e4655bc24942a726ed4efb0b3d67e"
server_key_iv: "da5cde3408efc710907f921e92301cc9"
server_key_version: "encryptionkey@0"
I20260927 22:09:07.269011 4050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:09:07.272392 4065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:07.273705 4050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:09:07.273872 4050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "33ae3061c0b54a80a4275b6999e5bda4"
format_stamp: "Formatted at 2026-09-27 22:09:07 on dist-test-slave-mp03"
server_key: "473e4655bc24942a726ed4efb0b3d67e"
server_key_iv: "da5cde3408efc710907f921e92301cc9"
server_key_version: "encryptionkey@0"
I20260927 22:09:07.274065 4050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:07.297073 4050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:07.301617 4050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:07.301923 4050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:07.302973 4050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:07.304318 4050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:07.304428 4050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:07.304572 4050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:07.304642 4050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:07.337345 4050 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40665
I20260927 22:09:07.337409 4177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40665 every 8 connection(s)
I20260927 22:09:07.338663 4050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:09:07.346426 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 4050
I20260927 22:09:07.346666 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:09:07.347429 388 external_mini_cluster.cc:1428] Setting key 6d146c7f960ebe005844fec59a99fc54
Sep 27 22:09:07 dist-test-slave-mp03 krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:09:07.358552 4180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:07.341264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55021 (local address 127.0.97.62:39933)
0927 22:09:07.341669 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:07.341674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:07.342729 (+ 1055us) server_negotiation.cc:407] Connection header received
0927 22:09:07.344507 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:07.344512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:07.344575 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:07.344670 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:07.346082 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:07.347261 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:07.348964 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:07.349352 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:07.349492 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:07.353090 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:07.353096 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:07.353099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:07.353129 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:07.355668 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:07.356361 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:07.356369 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:07.356377 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:07.356487 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:07.356897 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:07.356902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:07.356906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:07.357158 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:07.357305 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:07.357978 (+ 673us) server_negotiation.cc:299] Negotiation successful
0927 22:09:07.358188 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":211,"threads_started":1}
I20260927 22:09:07.359618 4181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:07.341860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39933 (local address 127.0.97.1:55021)
0927 22:09:07.342530 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:07.342574 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:07.344255 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:07.344875 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:07.344883 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:07.345375 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:07.345926 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:07.345942 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:07.347406 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:07.347410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:07.348804 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:07.348815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:07.349576 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:07.349585 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:07.349812 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:07.350499 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:07.350519 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:07.352860 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:07.355808 (+ 2948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:07.355818 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:07.355834 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:07.356236 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:07.356591 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:07.356597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:07.356601 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:07.356783 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:07.357278 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:07.357286 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:07.357623 (+ 337us) client_negotiation.cc:769] Sending connection context
0927 22:09:07.357888 (+ 265us) client_negotiation.cc:240] Negotiation successful
0927 22:09:07.358422 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":272,"threads_started":1}
I20260927 22:09:07.361380 4178 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39933
I20260927 22:09:07.361680 4178 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:07.362382 4178 heartbeater.cc:511] Master 127.0.97.62:39933 requested a full tablet report, sending...
I20260927 22:09:07.364835 3989 ts_manager.cc:194] Registered new tserver with Master: 33ae3061c0b54a80a4275b6999e5bda4 (127.0.97.1:40665)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:09:07.366447 3989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:09:07.390710 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39933
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:07.710913 4184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:07.711186 4184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:07.711287 4184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:07.718396 4184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:07.718515 4184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:07.718652 4184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:09:07.728443 4184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:07.729743 4184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:07.731098 4184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:07.744175 4192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:07.744220 4189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:07.747774 4190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:07.748188 4184 server_base.cc:1034] running on GCE node
I20260927 22:09:07.748769 4184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:07.749720 4184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:07.750944 4184 hybrid_clock.cc:648] HybridClock initialized: now 1790546947750893 us; error 42 us; skew 500 ppm
I20260927 22:09:07.751268 4184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:07 dist-test-slave-mp03 krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:07.754521 4184 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:09:07.756136 4184 webserver.cc:460] Webserver started at http://127.0.97.2:41035/ using document root <none> and password file <none>
I20260927 22:09:07.756882 4184 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:07.757028 4184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:07.757371 4184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:07.760412 4184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "0b075c1c629c4e4fb8a38a67edee15ef"
format_stamp: "Formatted at 2026-09-27 22:09:07 on dist-test-slave-mp03"
server_key: "4cc9d6a8065a9f9f4a87a561db0a1163"
server_key_iv: "23cd3f7734cd2c59bdfb55d27ee46926"
server_key_version: "encryptionkey@0"
I20260927 22:09:07.761191 4184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0b075c1c629c4e4fb8a38a67edee15ef"
format_stamp: "Formatted at 2026-09-27 22:09:07 on dist-test-slave-mp03"
server_key: "4cc9d6a8065a9f9f4a87a561db0a1163"
server_key_iv: "23cd3f7734cd2c59bdfb55d27ee46926"
server_key_version: "encryptionkey@0"
I20260927 22:09:07.766137 4184 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:09:07.769819 4199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:07.771137 4184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:09:07.771301 4184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "0b075c1c629c4e4fb8a38a67edee15ef"
format_stamp: "Formatted at 2026-09-27 22:09:07 on dist-test-slave-mp03"
server_key: "4cc9d6a8065a9f9f4a87a561db0a1163"
server_key_iv: "23cd3f7734cd2c59bdfb55d27ee46926"
server_key_version: "encryptionkey@0"
I20260927 22:09:07.771524 4184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:07.784988 4184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:07.789003 4184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:07.789243 4184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:07.790109 4184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:07.791426 4184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:07.791495 4184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:07.791574 4184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:07.791620 4184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:07.824261 4184 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35039
I20260927 22:09:07.824301 4311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35039 every 8 connection(s)
I20260927 22:09:07.825541 4184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:09:07.829085 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 4184
I20260927 22:09:07.829289 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:09:07.829756 388 external_mini_cluster.cc:1428] Setting key 66e3fc822c70b5b560ad8f4bf1203b49
Sep 27 22:09:07 dist-test-slave-mp03 krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:09:07.849184 4180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:07.828111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43603 (local address 127.0.97.62:39933)
0927 22:09:07.828321 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:07.828327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:07.829651 (+ 1324us) server_negotiation.cc:407] Connection header received
0927 22:09:07.831814 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:07.831819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:07.831892 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:07.831979 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:07.834126 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:07.834888 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:07.837375 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:07.837758 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:07.837891 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:07.841908 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:07.841917 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:07.841924 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:07.841966 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:07.845680 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:07.846612 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:07.846620 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:07.846626 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:07.846787 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:07.847211 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:07.847219 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:07.847225 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:07.847656 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:07.847816 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:07.848530 (+ 714us) server_negotiation.cc:299] Negotiation successful
0927 22:09:07.848787 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:09:07.849915 4314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:07.828611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39933 (local address 127.0.97.2:43603)
0927 22:09:07.829470 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:07.829522 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:07.831568 (+ 2046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:07.832257 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:07.832269 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:07.833051 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:07.833936 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:07.833965 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:07.835039 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:07.835047 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:07.837240 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:07.837257 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:07.837990 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:07.837998 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:07.838221 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:07.838996 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:07.839030 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:07.841679 (+ 2649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:07.845844 (+ 4165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:07.845854 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:07.845884 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:07.846466 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:07.846919 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:07.846922 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:07.846925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:07.847099 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:07.847783 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:07.847788 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:07.848148 (+ 360us) client_negotiation.cc:769] Sending connection context
0927 22:09:07.848379 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 22:09:07.848773 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":359,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:09:07.851526 4312 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39933
I20260927 22:09:07.851845 4312 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:07.852536 4312 heartbeater.cc:511] Master 127.0.97.62:39933 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 22:09:07.854238 3989 ts_manager.cc:194] Registered new tserver with Master: 0b075c1c629c4e4fb8a38a67edee15ef (127.0.97.2:35039)
I20260927 22:09:07.855135 3989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:09:07.878647 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39933
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:08.196646 4317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:08.197000 4317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:08.197154 4317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:08.204375 4317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:08.204530 4317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:08.204708 4317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:09:08.214636 4317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:08.216068 4317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:08.217869 4317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:08.231374 4322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:08.231498 4325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:08.231977 4323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:08.232422 4317 server_base.cc:1034] running on GCE node
I20260927 22:09:08.232963 4317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:08.233808 4317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:08.235036 4317 hybrid_clock.cc:648] HybridClock initialized: now 1790546948234988 us; error 69 us; skew 500 ppm
I20260927 22:09:08.235347 4317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:08 dist-test-slave-mp03 krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:08.238418 4317 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:09:08.239895 4317 webserver.cc:460] Webserver started at http://127.0.97.3:41117/ using document root <none> and password file <none>
I20260927 22:09:08.240567 4317 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:08.240707 4317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:08.241014 4317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:08.243950 4317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "150821500fd74baa926bb349d6d029b0"
format_stamp: "Formatted at 2026-09-27 22:09:08 on dist-test-slave-mp03"
server_key: "a7ca2608ce7ec8181d3dc984c925cd63"
server_key_iv: "941b446b72df63e6d9bde590131b8fb3"
server_key_version: "encryptionkey@0"
I20260927 22:09:08.244726 4317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "150821500fd74baa926bb349d6d029b0"
format_stamp: "Formatted at 2026-09-27 22:09:08 on dist-test-slave-mp03"
server_key: "a7ca2608ce7ec8181d3dc984c925cd63"
server_key_iv: "941b446b72df63e6d9bde590131b8fb3"
server_key_version: "encryptionkey@0"
I20260927 22:09:08.249876 4317 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260927 22:09:08.253347 4332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:08.254719 4317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 22:09:08.254889 4317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "150821500fd74baa926bb349d6d029b0"
format_stamp: "Formatted at 2026-09-27 22:09:08 on dist-test-slave-mp03"
server_key: "a7ca2608ce7ec8181d3dc984c925cd63"
server_key_iv: "941b446b72df63e6d9bde590131b8fb3"
server_key_version: "encryptionkey@0"
I20260927 22:09:08.255113 4317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:08.286289 4317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:08.290647 4317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:08.290954 4317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:08.291911 4317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:08.293251 4317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:08.293351 4317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:08.293502 4317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:08.293582 4317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:08.326040 4317 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41425
I20260927 22:09:08.326081 4444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41425 every 8 connection(s)
I20260927 22:09:08.327407 4317 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:09:08.327739 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 4317
I20260927 22:09:08.327948 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:09:08.328537 388 external_mini_cluster.cc:1428] Setting key 8de00c22e454e2323717e3aee30fe749
Sep 27 22:09:08 dist-test-slave-mp03 krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:09:08.348186 4180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:08.330232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34769 (local address 127.0.97.62:39933)
0927 22:09:08.330505 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:08.330509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:08.331863 (+ 1354us) server_negotiation.cc:407] Connection header received
0927 22:09:08.333672 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:08.333676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:08.333738 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:08.333821 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:08.335354 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:08.336085 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:08.338164 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:08.338554 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:08.338718 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:08.342789 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:08.342793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:08.342796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:08.342826 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:08.345574 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:08.346213 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:08.346217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:08.346219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:08.346330 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:08.346744 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:08.346747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:08.346749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:08.347035 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:08.347141 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:08.347758 (+ 617us) server_negotiation.cc:299] Negotiation successful
0927 22:09:08.347916 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260927 22:09:08.349119 4447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:08.330917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39933 (local address 127.0.97.3:34769)
0927 22:09:08.331728 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:08.331775 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:08.333416 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:08.334050 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:08.334058 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:08.334559 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:08.335210 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:08.335227 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:08.336213 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:08.336217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:08.338016 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:08.338033 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:08.338818 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:08.338825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:08.339123 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:08.339805 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:08.339834 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:08.342546 (+ 2712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:08.345693 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:08.345700 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:08.345712 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:08.346112 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:08.346415 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:08.346419 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:08.346421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:08.346590 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:08.347134 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:08.347140 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:08.347444 (+ 304us) client_negotiation.cc:769] Sending connection context
0927 22:09:08.347659 (+ 215us) client_negotiation.cc:240] Negotiation successful
0927 22:09:08.348023 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":365,"threads_started":1}
I20260927 22:09:08.350430 4445 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39933
I20260927 22:09:08.350772 4445 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:08.351414 4445 heartbeater.cc:511] Master 127.0.97.62:39933 requested a full tablet report, sending...
I20260927 22:09:08.352751 3989 ts_manager.cc:194] Registered new tserver with Master: 150821500fd74baa926bb349d6d029b0 (127.0.97.3:41425)
I20260927 22:09:08.353487 3989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34769
I20260927 22:09:08.356875 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:09:08.369597 4178 heartbeater.cc:503] Master 127.0.97.62:39933 was elected leader, sending a full tablet report...
Sep 27 22:09:08 dist-test-slave-mp03 krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:09:08 dist-test-slave-mp03 krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:09:08.405202 4452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:08.390239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38164 (local address 127.0.97.1:40665)
0927 22:09:08.390915 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:08.390925 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:08.390984 (+ 59us) server_negotiation.cc:407] Connection header received
0927 22:09:08.391065 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:08.391072 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:08.391292 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:08.391457 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:08.393241 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:08.394226 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:08.395515 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:08.396091 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:08.396244 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:08.400234 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:08.400245 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:08.400261 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:08.400295 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:08.402950 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:08.403518 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:08.403525 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:08.403527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:08.403635 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:08.403954 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:08.403957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:08.403959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:08.404266 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:08.404476 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:08.404722 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 22:09:08.404903 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":323,"threads_started":1}
Sep 27 22:09:08 dist-test-slave-mp03 krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:09:08.422205 4180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:08.409423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38270 (local address 127.0.97.62:39933)
0927 22:09:08.409661 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:08.409667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:08.409855 (+ 188us) server_negotiation.cc:407] Connection header received
0927 22:09:08.410020 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:08.410024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:08.410084 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:08.410154 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:08.411335 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:08.412020 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:08.413142 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:08.413534 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:08.413622 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:08.417305 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:08.417310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:08.417313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:08.417344 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:08.419697 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:08.420305 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:08.420312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:08.420317 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:08.420453 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:08.420806 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:08.420813 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:08.420816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:08.421122 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:08.421387 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:08.421682 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 22:09:08.421893 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 22:09:08.422856 3989 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38270
I20260927 22:09:08.423812 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 4050
I20260927 22:09:08.435797 4452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:08.424763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38168 (local address 127.0.97.1:40665)
0927 22:09:08.424958 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:08.424965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:08.425125 (+ 160us) server_negotiation.cc:407] Connection header received
0927 22:09:08.425198 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:08.425206 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:08.425322 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:08.425433 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:08.426488 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:08.427243 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:08.428394 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:08.428792 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:08.428883 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:08.430985 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:08.430990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:08.430993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:08.431022 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:08.433582 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:08.434094 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:08.434098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:08.434100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:08.434206 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:08.434502 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:08.434505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:08.434507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:08.434799 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:08.435073 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:08.435306 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 22:09:08.435465 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:09:08.436401 4173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:08.517191 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 4050
I20260927 22:09:08.533630 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 4184
Sep 27 22:09:08 dist-test-slave-mp03 krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:09:08.549090 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:08.534497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52004 (local address 127.0.97.2:35039)
0927 22:09:08.535141 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:08.535152 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:08.535184 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:09:08.535256 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:08.535263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:08.535487 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:08.535622 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:08.536721 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:08.537826 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:08.539069 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:08.539611 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:08.539705 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:08.543830 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:08.543842 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:08.543856 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:08.543890 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:08.546477 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:08.547279 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:08.547286 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:08.547288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:08.547407 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:08.547725 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:08.547728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:08.547730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:08.548033 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:08.548255 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:08.548529 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 22:09:08.548711 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":332,"threads_started":1}
I20260927 22:09:08.550338 4307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:08.632027 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 4184
I20260927 22:09:08.648568 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 4317
Sep 27 22:09:08 dist-test-slave-mp03 krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:09:08.664740 4459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:08.649577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50560 (local address 127.0.97.3:41425)
0927 22:09:08.650178 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:08.650185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:08.650214 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:09:08.650275 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:08.650281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:08.650491 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:08.650691 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:08.651917 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:08.652915 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:08.654152 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:08.654827 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:08.654988 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:08.659316 (+ 4328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:08.659332 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:08.659346 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:08.659380 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:08.661921 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:08.662562 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:08.662570 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:08.662574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:08.662826 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:08.663176 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:08.663184 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:08.663189 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:08.663525 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:08.663846 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:08.664137 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 22:09:08.664396 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":295,"threads_started":1}
I20260927 22:09:08.665978 4440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:08.747854 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 4317
I20260927 22:09:08.763428 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 3957
I20260927 22:09:08.764586 4019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:08.864816 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 3957
2026-09-27T22:09:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3817 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:09:08 dist-test-slave-mp03 krb5kdc[4464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:09:08 dist-test-slave-mp03 krb5kdc[4464](info): set up 2 sockets
Sep 27 22:09:08 dist-test-slave-mp03 krb5kdc[4464](info): commencing operation
krb5kdc: starting...
W20260927 22:09:09.959049 388 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.010s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:09:10 dist-test-slave-mp03 krb5kdc[4464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:09:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:09:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:09:10.154520 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36117
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:10.474005 4477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:10.474304 4477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:10.474466 4477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:10.481676 4477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:09:10.481789 4477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:10.481868 4477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:10.481954 4477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:09:10.482039 4477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:09:10.491959 4477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:10.493288 4477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:10.494669 4477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:10.506085 4485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:10.506117 4482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:10.506178 4483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:10.506678 4477 server_base.cc:1034] running on GCE node
I20260927 22:09:10.507511 4477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:10.508801 4477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:10.510003 4477 hybrid_clock.cc:648] HybridClock initialized: now 1790546950509966 us; error 33 us; skew 500 ppm
I20260927 22:09:10.510303 4477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:10 dist-test-slave-mp03 krb5kdc[4464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:10.513379 4477 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:09:10.514521 4477 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:10.514676 4477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:10.515022 4477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:10.517946 4477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "3647e7b9509c4048949df51ac809a350"
format_stamp: "Formatted at 2026-09-27 22:09:10 on dist-test-slave-mp03"
server_key: "bbb7965c50727a24fd83afec66051bdf"
server_key_iv: "f787cdfe92a4cb1f6144b19f698d9f31"
server_key_version: "encryptionkey@0"
I20260927 22:09:10.518846 4477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "3647e7b9509c4048949df51ac809a350"
format_stamp: "Formatted at 2026-09-27 22:09:10 on dist-test-slave-mp03"
server_key: "bbb7965c50727a24fd83afec66051bdf"
server_key_iv: "f787cdfe92a4cb1f6144b19f698d9f31"
server_key_version: "encryptionkey@0"
I20260927 22:09:10.523597 4477 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260927 22:09:10.526948 4491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:10.528216 4477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:09:10.528412 4477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "3647e7b9509c4048949df51ac809a350"
format_stamp: "Formatted at 2026-09-27 22:09:10 on dist-test-slave-mp03"
server_key: "bbb7965c50727a24fd83afec66051bdf"
server_key_iv: "f787cdfe92a4cb1f6144b19f698d9f31"
server_key_version: "encryptionkey@0"
I20260927 22:09:10.528628 4477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:10.545164 4477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:10.549080 4477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:10.549365 4477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:10.567209 4477 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36117
I20260927 22:09:10.567229 4542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36117 every 8 connection(s)
I20260927 22:09:10.568118 4477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:09:10.571601 4543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:10.572413 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 4477
I20260927 22:09:10.572638 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:09:10.573426 388 external_mini_cluster.cc:1428] Setting key 919dbc767a58500ed7a985c64c2f31f5
I20260927 22:09:10.583945 4543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3647e7b9509c4048949df51ac809a350: Bootstrap starting.
Sep 27 22:09:10 dist-test-slave-mp03 krb5kdc[4464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:09:10.587011 4543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3647e7b9509c4048949df51ac809a350: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:10.588233 4543 log.cc:824] T 00000000000000000000000000000000 P 3647e7b9509c4048949df51ac809a350: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:10.591302 4543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3647e7b9509c4048949df51ac809a350: No bootstrap required, opened a new log
I20260927 22:09:10.595489 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:10.575422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50750 (local address 127.0.97.62:36117)
0927 22:09:10.576138 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:10.576148 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:10.576188 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:09:10.577970 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:10.577993 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:10.578402 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:10.578870 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:10.580373 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:10.581653 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:10.582959 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:10.583612 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:10.583800 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:10.588011 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:10.588018 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:10.588032 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:10.588063 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:10.591453 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:10.592274 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:10.592280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:10.592286 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:10.592440 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:10.592832 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:10.592835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:10.592837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:10.593366 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:10.593586 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:10.594040 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 22:09:10.594435 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":314,"threads_started":1}
I20260927 22:09:10.599493 4543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3647e7b9509c4048949df51ac809a350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3647e7b9509c4048949df51ac809a350" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36117 } }
I20260927 22:09:10.599829 4543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3647e7b9509c4048949df51ac809a350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:10.599939 4543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3647e7b9509c4048949df51ac809a350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3647e7b9509c4048949df51ac809a350, State: Initialized, Role: FOLLOWER
I20260927 22:09:10.600531 4543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3647e7b9509c4048949df51ac809a350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3647e7b9509c4048949df51ac809a350" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36117 } }
I20260927 22:09:10.600746 4543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3647e7b9509c4048949df51ac809a350 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:09:10.600868 4543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3647e7b9509c4048949df51ac809a350 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:09:10.601025 4543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3647e7b9509c4048949df51ac809a350 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:10.603272 4543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3647e7b9509c4048949df51ac809a350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3647e7b9509c4048949df51ac809a350" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36117 } }
I20260927 22:09:10.603746 4543 leader_election.cc:302] T 00000000000000000000000000000000 P 3647e7b9509c4048949df51ac809a350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3647e7b9509c4048949df51ac809a350; no voters:
I20260927 22:09:10.604267 4543 leader_election.cc:288] T 00000000000000000000000000000000 P 3647e7b9509c4048949df51ac809a350 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:09:10.604420 4548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3647e7b9509c4048949df51ac809a350 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:09:10.605311 4548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3647e7b9509c4048949df51ac809a350 [term 1 LEADER]: Becoming Leader. State: Replica: 3647e7b9509c4048949df51ac809a350, State: Running, Role: LEADER
I20260927 22:09:10.605849 4548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3647e7b9509c4048949df51ac809a350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3647e7b9509c4048949df51ac809a350" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36117 } }
I20260927 22:09:10.606478 4543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3647e7b9509c4048949df51ac809a350 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:09:10.608767 4549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3647e7b9509c4048949df51ac809a350 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3647e7b9509c4048949df51ac809a350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3647e7b9509c4048949df51ac809a350" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36117 } } }
I20260927 22:09:10.609143 4549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3647e7b9509c4048949df51ac809a350 [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:10.608740 4550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3647e7b9509c4048949df51ac809a350 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3647e7b9509c4048949df51ac809a350. Latest consensus state: current_term: 1 leader_uuid: "3647e7b9509c4048949df51ac809a350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3647e7b9509c4048949df51ac809a350" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36117 } } }
I20260927 22:09:10.609561 4550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3647e7b9509c4048949df51ac809a350 [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:10.610314 4554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:09:10.616485 4554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:09:10.626021 4554 catalog_manager.cc:1347] Generated new cluster ID: f3a8b246027b46b9ab4ac85b6fa42a55
I20260927 22:09:10.626134 4554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:09:10.637943 4554 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:09:10.639276 4554 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:09:10.647500 4554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3647e7b9509c4048949df51ac809a350: Generated new TSK 0
I20260927 22:09:10.648350 4554 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:09:10.702162 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36117
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:11.057471 4569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:11.057811 4569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:11.057994 4569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:11.065258 4569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:11.065423 4569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:11.065605 4569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:09:11.076212 4569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:11.078047 4569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:11.079783 4569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:11.093303 4574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:11.093237 4575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:11.093765 4577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:11.094031 4569 server_base.cc:1034] running on GCE node
I20260927 22:09:11.094702 4569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:11.095587 4569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:11.096810 4569 hybrid_clock.cc:648] HybridClock initialized: now 1790546951096769 us; error 56 us; skew 500 ppm
I20260927 22:09:11.097122 4569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:11 dist-test-slave-mp03 krb5kdc[4464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:11.100252 4569 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:09:11.101450 4569 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:11.101593 4569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:11.101914 4569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:11.104754 4569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "1469e743139740d68e3c9519cf433db2"
format_stamp: "Formatted at 2026-09-27 22:09:11 on dist-test-slave-mp03"
server_key: "8d8670bf210c85333aeb001c70bf540a"
server_key_iv: "e1fe2adf7a166dcafdaa09d53c277047"
server_key_version: "encryptionkey@0"
I20260927 22:09:11.105536 4569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1469e743139740d68e3c9519cf433db2"
format_stamp: "Formatted at 2026-09-27 22:09:11 on dist-test-slave-mp03"
server_key: "8d8670bf210c85333aeb001c70bf540a"
server_key_iv: "e1fe2adf7a166dcafdaa09d53c277047"
server_key_version: "encryptionkey@0"
I20260927 22:09:11.110545 4569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260927 22:09:11.113919 4583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:11.115465 4569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:09:11.115628 4569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "1469e743139740d68e3c9519cf433db2"
format_stamp: "Formatted at 2026-09-27 22:09:11 on dist-test-slave-mp03"
server_key: "8d8670bf210c85333aeb001c70bf540a"
server_key_iv: "e1fe2adf7a166dcafdaa09d53c277047"
server_key_version: "encryptionkey@0"
I20260927 22:09:11.115841 4569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:11.137079 4569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:11.141444 4569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:11.141793 4569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:11.142872 4569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:11.144155 4569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:11.144245 4569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:11.144322 4569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:11.144387 4569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:11.178200 4569 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41013
I20260927 22:09:11.178222 4695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41013 every 8 connection(s)
I20260927 22:09:11.179168 4569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:09:11.183944 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 4569
I20260927 22:09:11.184154 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:09:11.184900 388 external_mini_cluster.cc:1428] Setting key a7ac5a950b26af1910c12a365a957e20
Sep 27 22:09:11 dist-test-slave-mp03 krb5kdc[4464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:09:11.200925 4698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:11.181588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36023 (local address 127.0.97.62:36117)
0927 22:09:11.182067 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:11.182071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:11.182879 (+ 808us) server_negotiation.cc:407] Connection header received
0927 22:09:11.185382 (+ 2503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:11.185387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:11.185460 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:11.185601 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:11.187471 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:11.188160 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:11.190731 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:11.191132 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:11.191226 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:11.195074 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:11.195079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:11.195082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:11.195114 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:11.197720 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:11.198486 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:11.198493 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:11.198501 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:11.198664 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:11.199093 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:11.199098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:11.199101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:11.199370 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:11.199516 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:11.200342 (+ 826us) server_negotiation.cc:299] Negotiation successful
0927 22:09:11.200558 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":285,"threads_started":1}
I20260927 22:09:11.201921 4699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:11.181995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36117 (local address 127.0.97.1:36023)
0927 22:09:11.182694 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:11.182738 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:11.185065 (+ 2327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:11.185854 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:11.185867 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:11.186440 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:11.187301 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:11.187327 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:11.188319 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:11.188328 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:11.190539 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:11.190557 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:11.191370 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:11.191380 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:11.191709 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:11.192464 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:11.192499 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:11.194833 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:11.197893 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:11.197902 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:11.197922 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:11.198354 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:11.198774 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:11.198779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:11.198783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:11.198975 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:11.199513 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:11.199522 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:11.199963 (+ 441us) client_negotiation.cc:769] Sending connection context
0927 22:09:11.200230 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 22:09:11.200775 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":268,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:09:11.203578 4696 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36117
I20260927 22:09:11.203873 4696 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:11.204514 4696 heartbeater.cc:511] Master 127.0.97.62:36117 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 22:09:11.206825 4508 ts_manager.cc:194] Registered new tserver with Master: 1469e743139740d68e3c9519cf433db2 (127.0.97.1:41013)
I20260927 22:09:11.208420 4508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:09:11.230818 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36117
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:11.552083 4702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:11.552394 4702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:11.552491 4702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:11.559969 4702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:11.560089 4702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:11.560205 4702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:09:11.572721 4702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:11.574393 4702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:11.576038 4702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:11.591089 4707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:11.591166 4710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:11.591161 4708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:11.592345 4702 server_base.cc:1034] running on GCE node
I20260927 22:09:11.592986 4702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:11.593945 4702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:11.595134 4702 hybrid_clock.cc:648] HybridClock initialized: now 1790546951595095 us; error 39 us; skew 500 ppm
I20260927 22:09:11.595453 4702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:11 dist-test-slave-mp03 krb5kdc[4464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:11.598714 4702 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:09:11.599934 4702 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:11.600023 4702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:11.600369 4702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:11.603302 4702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "e5a4e741ea4741c78d65de0d31c8f27d"
format_stamp: "Formatted at 2026-09-27 22:09:11 on dist-test-slave-mp03"
server_key: "f2a6b1442649106199c4c4bd10c19f08"
server_key_iv: "9387242f2bbc5d2df9dce2f4bd5e6fd7"
server_key_version: "encryptionkey@0"
I20260927 22:09:11.604100 4702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e5a4e741ea4741c78d65de0d31c8f27d"
format_stamp: "Formatted at 2026-09-27 22:09:11 on dist-test-slave-mp03"
server_key: "f2a6b1442649106199c4c4bd10c19f08"
server_key_iv: "9387242f2bbc5d2df9dce2f4bd5e6fd7"
server_key_version: "encryptionkey@0"
I20260927 22:09:11.609256 4702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:09:11.612677 4716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:11.614118 4702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:09:11.614274 4702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "e5a4e741ea4741c78d65de0d31c8f27d"
format_stamp: "Formatted at 2026-09-27 22:09:11 on dist-test-slave-mp03"
server_key: "f2a6b1442649106199c4c4bd10c19f08"
server_key_iv: "9387242f2bbc5d2df9dce2f4bd5e6fd7"
server_key_version: "encryptionkey@0"
I20260927 22:09:11.614495 4702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:11.631459 4702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:11.635813 4702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:11.636096 4702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:11.637050 4702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:11.638298 4702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:11.638379 4702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:11.638540 4702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:11.638626 4702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:11.672475 4702 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43373
I20260927 22:09:11.672546 4828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43373 every 8 connection(s)
I20260927 22:09:11.673316 4702 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:09:11.680095 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 4702
I20260927 22:09:11.680305 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:09:11.680845 388 external_mini_cluster.cc:1428] Setting key d88c9b6e0c633a4bb3eeee973aebb522
Sep 27 22:09:11 dist-test-slave-mp03 krb5kdc[4464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:09:11.701635 4698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:11.675737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53699 (local address 127.0.97.62:36117)
0927 22:09:11.675916 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:11.675921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:11.677080 (+ 1159us) server_negotiation.cc:407] Connection header received
0927 22:09:11.678881 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:11.678887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:11.678981 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:11.679093 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:11.680655 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:11.681474 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:11.683724 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:11.684141 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:11.684234 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:11.688347 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:11.688352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:11.688355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:11.688384 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:11.698279 (+ 9895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:11.699317 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:11.699324 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:11.699330 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:11.699498 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:11.700030 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:11.700033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:11.700036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:11.700285 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:11.700392 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:11.701229 (+ 837us) server_negotiation.cc:299] Negotiation successful
0927 22:09:11.701391 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 22:09:11.703088 4831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:11.676253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36117 (local address 127.0.97.2:53699)
0927 22:09:11.676902 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:11.676955 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:11.678541 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:11.679322 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:11.679330 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:11.679819 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:11.680488 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:11.680507 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:11.681798 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:11.681805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:11.683572 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:11.683591 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:11.684365 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:11.684375 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:11.684704 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:11.685462 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:11.685497 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:11.688124 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:11.698484 (+ 10360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:11.698493 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:11.698509 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:11.699163 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:11.699608 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:11.699611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:11.699613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:11.699898 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:11.700395 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:11.700401 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:11.700884 (+ 483us) client_negotiation.cc:769] Sending connection context
0927 22:09:11.701124 (+ 240us) client_negotiation.cc:240] Negotiation successful
0927 22:09:11.701612 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":261,"threads_started":1}
I20260927 22:09:11.704723 4829 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36117
I20260927 22:09:11.705066 4829 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:11.705780 4829 heartbeater.cc:511] Master 127.0.97.62:36117 requested a full tablet report, sending...
I20260927 22:09:11.707381 4508 ts_manager.cc:194] Registered new tserver with Master: e5a4e741ea4741c78d65de0d31c8f27d (127.0.97.2:43373)
I20260927 22:09:11.708334 4508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:09:11.729768 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36117
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:12.053589 4834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:12.054008 4834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:12.054180 4834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:12.061492 4834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:12.061638 4834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:12.061828 4834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:09:12.071664 4834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:12.073035 4834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:12.074312 4834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:12.087131 4840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:12.087366 4842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:12.087657 4839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:12.087939 4834 server_base.cc:1034] running on GCE node
I20260927 22:09:12.088567 4834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:12.089586 4834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:12.090821 4834 hybrid_clock.cc:648] HybridClock initialized: now 1790546952090762 us; error 50 us; skew 500 ppm
I20260927 22:09:12.091140 4834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:12 dist-test-slave-mp03 krb5kdc[4464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:12.094069 4834 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:09:12.095221 4834 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:12.095350 4834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:12.095664 4834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:12.098522 4834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "906b3bc5335a426cb582b74952b33941"
format_stamp: "Formatted at 2026-09-27 22:09:12 on dist-test-slave-mp03"
server_key: "d316f558459fff34048896db224ea365"
server_key_iv: "6f068352bd48b5b0fff02290a35764ac"
server_key_version: "encryptionkey@0"
I20260927 22:09:12.099362 4834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "906b3bc5335a426cb582b74952b33941"
format_stamp: "Formatted at 2026-09-27 22:09:12 on dist-test-slave-mp03"
server_key: "d316f558459fff34048896db224ea365"
server_key_iv: "6f068352bd48b5b0fff02290a35764ac"
server_key_version: "encryptionkey@0"
I20260927 22:09:12.104514 4834 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:09:12.108187 4848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:12.109546 4834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:09:12.109723 4834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "906b3bc5335a426cb582b74952b33941"
format_stamp: "Formatted at 2026-09-27 22:09:12 on dist-test-slave-mp03"
server_key: "d316f558459fff34048896db224ea365"
server_key_iv: "6f068352bd48b5b0fff02290a35764ac"
server_key_version: "encryptionkey@0"
I20260927 22:09:12.109946 4834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:12.133059 4834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:12.137183 4834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:12.137447 4834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:12.138423 4834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:12.139741 4834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:12.139865 4834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:12.139997 4834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:12.140084 4834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:12.172588 4834 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45845
I20260927 22:09:12.172606 4960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45845 every 8 connection(s)
I20260927 22:09:12.173445 4834 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:09:12.177999 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 4834
I20260927 22:09:12.178198 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:09:12.178668 388 external_mini_cluster.cc:1428] Setting key f93cdf726fb5d51e2ea2bcf10864894f
Sep 27 22:09:12 dist-test-slave-mp03 krb5kdc[4464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:09:12.193137 4698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:12.175817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54487 (local address 127.0.97.62:36117)
0927 22:09:12.175992 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:12.175996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:12.177066 (+ 1070us) server_negotiation.cc:407] Connection header received
0927 22:09:12.179584 (+ 2518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:12.179590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:12.179672 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:12.179788 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:12.181601 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.182412 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.183938 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.184342 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.184470 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:12.187819 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:12.187833 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:12.187836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:12.187865 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:12.190143 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:12.190936 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:12.190941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:12.190945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:12.191074 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:12.191474 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:12.191479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:12.191482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:12.191745 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:12.191897 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:12.192592 (+ 695us) server_negotiation.cc:299] Negotiation successful
0927 22:09:12.192807 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:09:12.194254 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:12.176241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36117 (local address 127.0.97.3:54487)
0927 22:09:12.176902 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:12.176948 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:12.179299 (+ 2351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:12.180019 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:12.180027 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:12.180752 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:12.181421 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:12.181438 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.182565 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.182570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:12.183797 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:12.183808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.184553 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.184562 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:12.184795 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:12.185388 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:12.185455 (+ 67us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:12.187607 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:12.190283 (+ 2676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:12.190294 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:12.190346 (+ 52us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:12.190818 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:12.191188 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:12.191193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:12.191196 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:12.191358 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:12.191871 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:12.191878 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:12.192197 (+ 319us) client_negotiation.cc:769] Sending connection context
0927 22:09:12.192459 (+ 262us) client_negotiation.cc:240] Negotiation successful
0927 22:09:12.192969 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":255,"threads_started":1}
I20260927 22:09:12.195616 4961 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36117
I20260927 22:09:12.195950 4961 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:12.196602 4961 heartbeater.cc:511] Master 127.0.97.62:36117 requested a full tablet report, sending...
I20260927 22:09:12.197958 4508 ts_manager.cc:194] Registered new tserver with Master: 906b3bc5335a426cb582b74952b33941 (127.0.97.3:45845)
I20260927 22:09:12.198818 4508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54487
I20260927 22:09:12.206465 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:09:12.211333 4696 heartbeater.cc:503] Master 127.0.97.62:36117 was elected leader, sending a full tablet report...
I20260927 22:09:12.223560 4698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:12.210735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50766 (local address 127.0.97.62:36117)
0927 22:09:12.210891 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:12.210896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:12.211358 (+ 462us) server_negotiation.cc:407] Connection header received
0927 22:09:12.211585 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:12.211590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:12.211680 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:12.211761 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:12.213581 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.214282 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.215668 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.216079 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.216175 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:12.218784 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:12.218789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:12.218792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:12.218819 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:12.221209 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:12.221794 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:12.221798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:12.221800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:12.221906 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:12.222210 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:12.222213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:12.222215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:12.222502 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:12.222733 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:12.223070 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 22:09:12.223224 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:09:12.227406 4508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:09:12.235025 4508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:09:12.264117 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:12.246756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43373 (local address 127.0.0.1:56632)
0927 22:09:12.255029 (+ 8273us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:12.255074 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:12.255238 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:12.257100 (+ 1862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:12.257111 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:12.257156 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:09:12.257642 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:12.257655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.259865 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.259870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:12.261561 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:12.261572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.263096 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.263102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:12.263288 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:12.263343 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 22:09:12.263449 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 22:09:12.263629 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8017,"mutex_wait_us":40,"spinlock_wait_cycles":6144,"thread_start_us":7872,"threads_started":1}
I20260927 22:09:12.264850 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:12.246798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45845 (local address 127.0.0.1:48420)
0927 22:09:12.255780 (+ 8982us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:12.255799 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:12.255920 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:12.256904 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:12.256911 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:12.256969 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:09:12.257500 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:12.257514 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.259520 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.259525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:12.262039 (+ 2514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:12.262053 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.263842 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.263848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:12.264050 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:12.264093 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 22:09:12.264311 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 22:09:12.264443 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8871,"mutex_wait_us":22,"spinlock_wait_cycles":12032,"thread_start_us":8748,"threads_started":1}
I20260927 22:09:12.265070 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:12.247379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48420 (local address 127.0.97.3:45845)
0927 22:09:12.248177 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:12.248190 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:12.255929 (+ 7739us) server_negotiation.cc:407] Connection header received
0927 22:09:12.256072 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:12.256084 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:12.256378 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:12.256602 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:09:12.257666 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.259350 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.262216 (+ 2866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.263665 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.263780 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:12.264025 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:12.264417 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 22:09:12.264604 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":432,"threads_started":1}
I20260927 22:09:12.265106 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:12.246826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56632 (local address 127.0.97.2:43373)
0927 22:09:12.256457 (+ 9631us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:12.256470 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:12.256503 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:09:12.256598 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:12.256605 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:12.256844 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:12.258045 (+ 1201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:09:12.258261 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.259723 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.261717 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.262866 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.264368 (+ 1502us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:12.264552 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:12.264621 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 22:09:12.264782 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9440,"thread_start_us":9325,"threads_started":1}
I20260927 22:09:12.265406 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:12.255837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38468 (local address 127.0.97.1:41013)
0927 22:09:12.256786 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:12.256794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:12.257548 (+ 754us) server_negotiation.cc:407] Connection header received
0927 22:09:12.257764 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:12.257770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:12.257918 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:12.258128 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:09:12.258940 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.259972 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.262467 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.263907 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.264027 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:12.264274 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:12.264943 (+ 669us) server_negotiation.cc:299] Negotiation successful
0927 22:09:12.265099 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":631,"threads_started":1}
I20260927 22:09:12.265612 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:12.256781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41013 (local address 127.0.0.1:38468)
0927 22:09:12.257406 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:12.257431 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:12.257585 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:12.258360 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:12.258364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:12.258384 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:09:12.258771 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:12.258784 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.260134 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.260145 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:12.262316 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:12.262328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.264406 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.264411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:12.264584 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:12.264615 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:09:12.264703 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 22:09:12.264825 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":315,"threads_started":1}
I20260927 22:09:12.269071 4764 tablet_service.cc:1467] Processing CreateTablet for tablet c30d33981f424969a6c0cecc1cb17e47 (DEFAULT_TABLE table=test-table [id=569545ad1a0449578f247c1912a580eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:09:12.269069 4896 tablet_service.cc:1467] Processing CreateTablet for tablet c30d33981f424969a6c0cecc1cb17e47 (DEFAULT_TABLE table=test-table [id=569545ad1a0449578f247c1912a580eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:09:12.270548 4764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c30d33981f424969a6c0cecc1cb17e47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:12.270539 4896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c30d33981f424969a6c0cecc1cb17e47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:12.271149 4631 tablet_service.cc:1467] Processing CreateTablet for tablet c30d33981f424969a6c0cecc1cb17e47 (DEFAULT_TABLE table=test-table [id=569545ad1a0449578f247c1912a580eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:09:12.272275 4631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c30d33981f424969a6c0cecc1cb17e47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:12.283339 4979 tablet_bootstrap.cc:492] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2: Bootstrap starting.
I20260927 22:09:12.285231 4980 tablet_bootstrap.cc:492] T c30d33981f424969a6c0cecc1cb17e47 P e5a4e741ea4741c78d65de0d31c8f27d: Bootstrap starting.
I20260927 22:09:12.288904 4979 tablet_bootstrap.cc:654] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:12.288904 4980 tablet_bootstrap.cc:654] T c30d33981f424969a6c0cecc1cb17e47 P e5a4e741ea4741c78d65de0d31c8f27d: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:12.289798 4981 tablet_bootstrap.cc:492] T c30d33981f424969a6c0cecc1cb17e47 P 906b3bc5335a426cb582b74952b33941: Bootstrap starting.
I20260927 22:09:12.290196 4980 log.cc:824] T c30d33981f424969a6c0cecc1cb17e47 P e5a4e741ea4741c78d65de0d31c8f27d: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:12.290896 4979 log.cc:824] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:12.292848 4981 tablet_bootstrap.cc:654] T c30d33981f424969a6c0cecc1cb17e47 P 906b3bc5335a426cb582b74952b33941: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:12.293666 4981 log.cc:824] T c30d33981f424969a6c0cecc1cb17e47 P 906b3bc5335a426cb582b74952b33941: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:12.293779 4980 tablet_bootstrap.cc:492] T c30d33981f424969a6c0cecc1cb17e47 P e5a4e741ea4741c78d65de0d31c8f27d: No bootstrap required, opened a new log
I20260927 22:09:12.294118 4980 ts_tablet_manager.cc:1397] T c30d33981f424969a6c0cecc1cb17e47 P e5a4e741ea4741c78d65de0d31c8f27d: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260927 22:09:12.297112 4981 tablet_bootstrap.cc:492] T c30d33981f424969a6c0cecc1cb17e47 P 906b3bc5335a426cb582b74952b33941: No bootstrap required, opened a new log
I20260927 22:09:12.297365 4981 ts_tablet_manager.cc:1397] T c30d33981f424969a6c0cecc1cb17e47 P 906b3bc5335a426cb582b74952b33941: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 22:09:12.297717 4979 tablet_bootstrap.cc:492] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2: No bootstrap required, opened a new log
I20260927 22:09:12.297966 4979 ts_tablet_manager.cc:1397] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20260927 22:09:12.304354 4981 raft_consensus.cc:348] T c30d33981f424969a6c0cecc1cb17e47 P 906b3bc5335a426cb582b74952b33941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5a4e741ea4741c78d65de0d31c8f27d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43373 } } peers { permanent_uuid: "906b3bc5335a426cb582b74952b33941" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45845 } } peers { permanent_uuid: "1469e743139740d68e3c9519cf433db2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41013 } }
I20260927 22:09:12.304740 4981 raft_consensus.cc:374] T c30d33981f424969a6c0cecc1cb17e47 P 906b3bc5335a426cb582b74952b33941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:12.304833 4981 raft_consensus.cc:729] T c30d33981f424969a6c0cecc1cb17e47 P 906b3bc5335a426cb582b74952b33941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 906b3bc5335a426cb582b74952b33941, State: Initialized, Role: FOLLOWER
I20260927 22:09:12.304654 4980 raft_consensus.cc:348] T c30d33981f424969a6c0cecc1cb17e47 P e5a4e741ea4741c78d65de0d31c8f27d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5a4e741ea4741c78d65de0d31c8f27d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43373 } } peers { permanent_uuid: "906b3bc5335a426cb582b74952b33941" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45845 } } peers { permanent_uuid: "1469e743139740d68e3c9519cf433db2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41013 } }
I20260927 22:09:12.305085 4980 raft_consensus.cc:374] T c30d33981f424969a6c0cecc1cb17e47 P e5a4e741ea4741c78d65de0d31c8f27d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:12.305182 4980 raft_consensus.cc:729] T c30d33981f424969a6c0cecc1cb17e47 P e5a4e741ea4741c78d65de0d31c8f27d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5a4e741ea4741c78d65de0d31c8f27d, State: Initialized, Role: FOLLOWER
I20260927 22:09:12.305560 4981 consensus_queue.cc:260] T c30d33981f424969a6c0cecc1cb17e47 P 906b3bc5335a426cb582b74952b33941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5a4e741ea4741c78d65de0d31c8f27d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43373 } } peers { permanent_uuid: "906b3bc5335a426cb582b74952b33941" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45845 } } peers { permanent_uuid: "1469e743139740d68e3c9519cf433db2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41013 } }
I20260927 22:09:12.305721 4980 consensus_queue.cc:260] T c30d33981f424969a6c0cecc1cb17e47 P e5a4e741ea4741c78d65de0d31c8f27d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5a4e741ea4741c78d65de0d31c8f27d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43373 } } peers { permanent_uuid: "906b3bc5335a426cb582b74952b33941" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45845 } } peers { permanent_uuid: "1469e743139740d68e3c9519cf433db2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41013 } }
I20260927 22:09:12.307036 4829 heartbeater.cc:503] Master 127.0.97.62:36117 was elected leader, sending a full tablet report...
I20260927 22:09:12.307603 4980 ts_tablet_manager.cc:1428] T c30d33981f424969a6c0cecc1cb17e47 P e5a4e741ea4741c78d65de0d31c8f27d: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260927 22:09:12.309334 4979 raft_consensus.cc:348] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5a4e741ea4741c78d65de0d31c8f27d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43373 } } peers { permanent_uuid: "906b3bc5335a426cb582b74952b33941" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45845 } } peers { permanent_uuid: "1469e743139740d68e3c9519cf433db2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41013 } }
I20260927 22:09:12.309747 4979 raft_consensus.cc:374] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:12.309849 4979 raft_consensus.cc:729] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1469e743139740d68e3c9519cf433db2, State: Initialized, Role: FOLLOWER
I20260927 22:09:12.310506 4979 consensus_queue.cc:260] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5a4e741ea4741c78d65de0d31c8f27d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43373 } } peers { permanent_uuid: "906b3bc5335a426cb582b74952b33941" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45845 } } peers { permanent_uuid: "1469e743139740d68e3c9519cf433db2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41013 } }
I20260927 22:09:12.312536 4979 ts_tablet_manager.cc:1428] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260927 22:09:12.314088 4961 heartbeater.cc:503] Master 127.0.97.62:36117 was elected leader, sending a full tablet report...
I20260927 22:09:12.314167 4981 ts_tablet_manager.cc:1428] T c30d33981f424969a6c0cecc1cb17e47 P 906b3bc5335a426cb582b74952b33941: Time spent starting tablet: real 0.017s user 0.004s sys 0.013s
W20260927 22:09:12.425045 4830 tablet.cc:2369] T c30d33981f424969a6c0cecc1cb17e47 P e5a4e741ea4741c78d65de0d31c8f27d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:09:12.425037 4962 tablet.cc:2369] T c30d33981f424969a6c0cecc1cb17e47 P 906b3bc5335a426cb582b74952b33941: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:09:12.431334 4697 tablet.cc:2369] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:09:12.476327 4987 raft_consensus.cc:482] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:09:12.476693 4987 raft_consensus.cc:504] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5a4e741ea4741c78d65de0d31c8f27d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43373 } } peers { permanent_uuid: "906b3bc5335a426cb582b74952b33941" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45845 } } peers { permanent_uuid: "1469e743139740d68e3c9519cf433db2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41013 } }
I20260927 22:09:12.478391 4987 leader_election.cc:288] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5a4e741ea4741c78d65de0d31c8f27d (127.0.97.2:43373), 906b3bc5335a426cb582b74952b33941 (127.0.97.3:45845)
I20260927 22:09:12.487775 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:12.478842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37409 (local address 127.0.97.3:45845)
0927 22:09:12.479046 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:12.479052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:12.479828 (+ 776us) server_negotiation.cc:407] Connection header received
0927 22:09:12.480008 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:12.480013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:12.480092 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:12.480193 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:09:12.481197 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.482263 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.485261 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.486550 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.486705 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:12.486807 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:12.487380 (+ 573us) server_negotiation.cc:299] Negotiation successful
0927 22:09:12.487511 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:09:12.487775 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:12.478849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42517 (local address 127.0.97.2:43373)
0927 22:09:12.479046 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:12.479052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:12.479820 (+ 768us) server_negotiation.cc:407] Connection header received
0927 22:09:12.480008 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:12.480013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:12.480092 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:12.480193 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:09:12.481155 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.482263 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.485307 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.486562 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.486705 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:12.486805 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:12.487380 (+ 575us) server_negotiation.cc:299] Negotiation successful
0927 22:09:12.487511 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:09:12.488093 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:12.478757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43373 (local address 127.0.97.1:42517)
0927 22:09:12.479649 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:12.479670 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:12.479857 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:12.480228 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:12.480234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:12.480269 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:09:12.480969 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:12.480983 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.482406 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.482413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:12.485027 (+ 2614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:12.485043 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.486854 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.486861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:12.487119 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:12.487239 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 22:09:12.487339 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 22:09:12.487591 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":300,"spinlock_wait_cycles":23040,"thread_start_us":479,"threads_started":1}
I20260927 22:09:12.488423 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:12.478831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45845 (local address 127.0.97.1:37409)
0927 22:09:12.479705 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:12.479727 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:12.479857 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:12.480228 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:12.480234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:12.480266 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:09:12.481041 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:12.481054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.482406 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.482413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:12.485027 (+ 2614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:12.485043 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.486843 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.486850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:12.487119 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:12.487241 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 22:09:12.487339 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 22:09:12.487591 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":355,"spinlock_wait_cycles":16768,"thread_start_us":445,"threads_started":1}
I20260927 22:09:12.489104 4784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30d33981f424969a6c0cecc1cb17e47" candidate_uuid: "1469e743139740d68e3c9519cf433db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5a4e741ea4741c78d65de0d31c8f27d" is_pre_election: true
I20260927 22:09:12.489324 4916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30d33981f424969a6c0cecc1cb17e47" candidate_uuid: "1469e743139740d68e3c9519cf433db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "906b3bc5335a426cb582b74952b33941" is_pre_election: true
I20260927 22:09:12.489645 4784 raft_consensus.cc:2454] T c30d33981f424969a6c0cecc1cb17e47 P e5a4e741ea4741c78d65de0d31c8f27d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1469e743139740d68e3c9519cf433db2 in term 0.
I20260927 22:09:12.489816 4916 raft_consensus.cc:2454] T c30d33981f424969a6c0cecc1cb17e47 P 906b3bc5335a426cb582b74952b33941 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1469e743139740d68e3c9519cf433db2 in term 0.
I20260927 22:09:12.490360 4584 leader_election.cc:302] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1469e743139740d68e3c9519cf433db2, e5a4e741ea4741c78d65de0d31c8f27d; no voters:
I20260927 22:09:12.490795 4987 raft_consensus.cc:2781] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:09:12.490942 4987 raft_consensus.cc:482] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:09:12.491061 4987 raft_consensus.cc:3037] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:12.493171 4987 raft_consensus.cc:504] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5a4e741ea4741c78d65de0d31c8f27d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43373 } } peers { permanent_uuid: "906b3bc5335a426cb582b74952b33941" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45845 } } peers { permanent_uuid: "1469e743139740d68e3c9519cf433db2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41013 } }
I20260927 22:09:12.493778 4987 leader_election.cc:288] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2 [CANDIDATE]: Term 1 election: Requested vote from peers e5a4e741ea4741c78d65de0d31c8f27d (127.0.97.2:43373), 906b3bc5335a426cb582b74952b33941 (127.0.97.3:45845)
I20260927 22:09:12.494213 4784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30d33981f424969a6c0cecc1cb17e47" candidate_uuid: "1469e743139740d68e3c9519cf433db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5a4e741ea4741c78d65de0d31c8f27d"
I20260927 22:09:12.494232 4916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30d33981f424969a6c0cecc1cb17e47" candidate_uuid: "1469e743139740d68e3c9519cf433db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "906b3bc5335a426cb582b74952b33941"
I20260927 22:09:12.494421 4916 raft_consensus.cc:3037] T c30d33981f424969a6c0cecc1cb17e47 P 906b3bc5335a426cb582b74952b33941 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:12.494414 4784 raft_consensus.cc:3037] T c30d33981f424969a6c0cecc1cb17e47 P e5a4e741ea4741c78d65de0d31c8f27d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:12.496481 4916 raft_consensus.cc:2454] T c30d33981f424969a6c0cecc1cb17e47 P 906b3bc5335a426cb582b74952b33941 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1469e743139740d68e3c9519cf433db2 in term 1.
I20260927 22:09:12.496522 4784 raft_consensus.cc:2454] T c30d33981f424969a6c0cecc1cb17e47 P e5a4e741ea4741c78d65de0d31c8f27d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1469e743139740d68e3c9519cf433db2 in term 1.
I20260927 22:09:12.496891 4587 leader_election.cc:302] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1469e743139740d68e3c9519cf433db2, 906b3bc5335a426cb582b74952b33941; no voters:
I20260927 22:09:12.497186 4987 raft_consensus.cc:2781] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:09:12.497610 4987 raft_consensus.cc:686] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2 [term 1 LEADER]: Becoming Leader. State: Replica: 1469e743139740d68e3c9519cf433db2, State: Running, Role: LEADER
I20260927 22:09:12.498143 4987 consensus_queue.cc:237] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5a4e741ea4741c78d65de0d31c8f27d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43373 } } peers { permanent_uuid: "906b3bc5335a426cb582b74952b33941" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45845 } } peers { permanent_uuid: "1469e743139740d68e3c9519cf433db2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41013 } }
I20260927 22:09:12.502393 4508 catalog_manager.cc:5495] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1469e743139740d68e3c9519cf433db2 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "1469e743139740d68e3c9519cf433db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5a4e741ea4741c78d65de0d31c8f27d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "906b3bc5335a426cb582b74952b33941" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1469e743139740d68e3c9519cf433db2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41013 } health_report { overall_health: HEALTHY } } }
I20260927 22:09:12.553833 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:12.549154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38470 (local address 127.0.97.1:41013)
0927 22:09:12.549307 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:12.549312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:12.549328 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:09:12.549417 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:12.549420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:12.549493 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:12.549626 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:09:12.549926 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.550682 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.551960 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.552356 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.552446 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:12.552884 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:09:12.553392 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:09:12.553503 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:12.553574 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 22:09:12.553637 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 22:09:12.561682 4784 raft_consensus.cc:1261] T c30d33981f424969a6c0cecc1cb17e47 P e5a4e741ea4741c78d65de0d31c8f27d [term 1 FOLLOWER]: Refusing update from remote peer 1469e743139740d68e3c9519cf433db2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:09:12.561683 4916 raft_consensus.cc:1261] T c30d33981f424969a6c0cecc1cb17e47 P 906b3bc5335a426cb582b74952b33941 [term 1 FOLLOWER]: Refusing update from remote peer 1469e743139740d68e3c9519cf433db2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:09:12.562695 4990 consensus_queue.cc:1035] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "906b3bc5335a426cb582b74952b33941" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:09:12.563026 4987 consensus_queue.cc:1035] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5a4e741ea4741c78d65de0d31c8f27d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:09:12.577939 4993 mvcc.cc:205] Tried to move back new op lower bound from 7334080317683343360 to 7334080317435879424. Current Snapshot: MvccSnapshot[applied={T|T < 7334080317683343360}]
I20260927 22:09:12.600869 4508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:09:12.601137 4508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:09:12.605043 4508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3647e7b9509c4048949df51ac809a350: Sending DeleteTablet for 3 replicas of tablet c30d33981f424969a6c0cecc1cb17e47
I20260927 22:09:12.606220 4631 tablet_service.cc:1514] Processing DeleteTablet for tablet c30d33981f424969a6c0cecc1cb17e47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38468
I20260927 22:09:12.606263 4896 tablet_service.cc:1514] Processing DeleteTablet for tablet c30d33981f424969a6c0cecc1cb17e47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48420
I20260927 22:09:12.606973 4764 tablet_service.cc:1514] Processing DeleteTablet for tablet c30d33981f424969a6c0cecc1cb17e47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56632
I20260927 22:09:12.606984 4630 tablet_service.cc:1514] Processing DeleteTablet for tablet c30d33981f424969a6c0cecc1cb17e47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38468
I20260927 22:09:12.607349 5003 tablet_replica.cc:330] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2: stopping tablet replica
I20260927 22:09:12.607913 5003 raft_consensus.cc:2229] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:09:12.608747 5003 raft_consensus.cc:2258] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:09:12.609170 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 4569
I20260927 22:09:12.615764 5003 ts_tablet_manager.cc:1905] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:12.619196 5004 tablet_replica.cc:330] T c30d33981f424969a6c0cecc1cb17e47 P 906b3bc5335a426cb582b74952b33941: stopping tablet replica
I20260927 22:09:12.619801 5004 raft_consensus.cc:2229] T c30d33981f424969a6c0cecc1cb17e47 P 906b3bc5335a426cb582b74952b33941 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:09:12.620453 5004 raft_consensus.cc:2258] T c30d33981f424969a6c0cecc1cb17e47 P 906b3bc5335a426cb582b74952b33941 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:09:12.622936 5004 ts_tablet_manager.cc:1905] T c30d33981f424969a6c0cecc1cb17e47 P 906b3bc5335a426cb582b74952b33941: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:12.627276 5005 tablet_replica.cc:330] T c30d33981f424969a6c0cecc1cb17e47 P e5a4e741ea4741c78d65de0d31c8f27d: stopping tablet replica
I20260927 22:09:12.627743 5005 raft_consensus.cc:2229] T c30d33981f424969a6c0cecc1cb17e47 P e5a4e741ea4741c78d65de0d31c8f27d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:09:12.628268 5005 raft_consensus.cc:2258] T c30d33981f424969a6c0cecc1cb17e47 P e5a4e741ea4741c78d65de0d31c8f27d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:09:12.628993 5003 ts_tablet_manager.cc:1918] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:09:12.629173 5003 log.cc:1196] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-0/wal/wals/c30d33981f424969a6c0cecc1cb17e47
I20260927 22:09:12.629776 5003 ts_tablet_manager.cc:1939] T c30d33981f424969a6c0cecc1cb17e47 P 1469e743139740d68e3c9519cf433db2: Deleting consensus metadata
I20260927 22:09:12.630787 5004 ts_tablet_manager.cc:1918] T c30d33981f424969a6c0cecc1cb17e47 P 906b3bc5335a426cb582b74952b33941: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:09:12.631062 5004 log.cc:1196] T c30d33981f424969a6c0cecc1cb17e47 P 906b3bc5335a426cb582b74952b33941: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/c30d33981f424969a6c0cecc1cb17e47
I20260927 22:09:12.631772 5004 ts_tablet_manager.cc:1939] T c30d33981f424969a6c0cecc1cb17e47 P 906b3bc5335a426cb582b74952b33941: Deleting consensus metadata
I20260927 22:09:12.632006 4492 catalog_manager.cc:4841] TS 1469e743139740d68e3c9519cf433db2 (127.0.97.1:41013): tablet c30d33981f424969a6c0cecc1cb17e47 (table test-table [id=569545ad1a0449578f247c1912a580eb]) successfully deleted
Sep 27 22:09:12 dist-test-slave-mp03 krb5kdc[4464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
W20260927 22:09:12.633169 4492 catalog_manager.cc:4808] TS 1469e743139740d68e3c9519cf433db2 (127.0.97.1:41013): delete failed for tablet c30d33981f424969a6c0cecc1cb17e47 because the tablet was not found. No further retry: Not found: Tablet not found: c30d33981f424969a6c0cecc1cb17e47
I20260927 22:09:12.634007 4495 catalog_manager.cc:4841] TS 906b3bc5335a426cb582b74952b33941 (127.0.97.3:45845): tablet c30d33981f424969a6c0cecc1cb17e47 (table test-table [id=569545ad1a0449578f247c1912a580eb]) successfully deleted
I20260927 22:09:12.635012 5005 ts_tablet_manager.cc:1905] T c30d33981f424969a6c0cecc1cb17e47 P e5a4e741ea4741c78d65de0d31c8f27d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:12.639639 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:12.610942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38472 (local address 127.0.97.1:41013)
0927 22:09:12.611525 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:12.611533 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:12.621481 (+ 9948us) server_negotiation.cc:407] Connection header received
0927 22:09:12.621703 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:12.621708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:12.621799 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:12.621911 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:12.623650 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.624941 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.626806 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.627588 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.627685 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:12.633284 (+ 5599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:12.633297 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:12.633327 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:12.633376 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:12.636412 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:12.637253 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:12.637262 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:12.637266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:12.637448 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:12.637901 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:12.637906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:12.637910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:12.638348 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:12.638680 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:12.639013 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 22:09:12.639255 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438}
I20260927 22:09:12.640306 4691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:12.640735 5005 ts_tablet_manager.cc:1918] T c30d33981f424969a6c0cecc1cb17e47 P e5a4e741ea4741c78d65de0d31c8f27d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:09:12.640926 5005 log.cc:1196] T c30d33981f424969a6c0cecc1cb17e47 P e5a4e741ea4741c78d65de0d31c8f27d: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/c30d33981f424969a6c0cecc1cb17e47
I20260927 22:09:12.641502 5005 ts_tablet_manager.cc:1939] T c30d33981f424969a6c0cecc1cb17e47 P e5a4e741ea4741c78d65de0d31c8f27d: Deleting consensus metadata
I20260927 22:09:12.643471 4492 catalog_manager.cc:4841] TS e5a4e741ea4741c78d65de0d31c8f27d (127.0.97.2:43373): tablet c30d33981f424969a6c0cecc1cb17e47 (table test-table [id=569545ad1a0449578f247c1912a580eb]) successfully deleted
I20260927 22:09:12.737797 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 4569
I20260927 22:09:12.755103 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 4702
Sep 27 22:09:12 dist-test-slave-mp03 krb5kdc[4464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:09:12.769881 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:12.755843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56642 (local address 127.0.97.2:43373)
0927 22:09:12.756048 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:12.756055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:12.756078 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:09:12.756287 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:12.756292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:12.756362 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:12.756468 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:12.757706 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.758452 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.759703 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.760102 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.760228 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:12.764439 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:12.764453 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:12.764470 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:12.764514 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:12.767438 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:12.768146 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:12.768152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:12.768155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:12.768280 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:12.768639 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:12.768642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:12.768644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:12.768995 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:12.769203 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:12.769481 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 22:09:12.769635 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:09:12.770654 4824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:12.863461 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 4702
I20260927 22:09:12.880900 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 4834
Sep 27 22:09:12 dist-test-slave-mp03 krb5kdc[4464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:09:12.896054 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:12.881730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48424 (local address 127.0.97.3:45845)
0927 22:09:12.881955 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:12.881961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:12.881982 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:09:12.882169 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:12.882173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:12.882256 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:12.882372 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:12.883726 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.884463 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.885713 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:12.886143 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:12.886276 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:12.890503 (+ 4227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:12.890519 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:12.890535 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:12.890570 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:12.893254 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:12.893902 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:12.893910 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:12.893912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:12.894039 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:12.894433 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:12.894442 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:12.894444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:12.894897 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:12.895202 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:12.895490 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 22:09:12.895709 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 22:09:12.896767 4956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:12.985996 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 4834
I20260927 22:09:13.003721 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 4477
I20260927 22:09:13.004890 4538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:13.117237 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 4477
2026-09-27T22:09:13Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4253 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T22:09:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:09:13Z Disabled control of system clock
I20260927 22:09:13.186363 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35229
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 22:09:13.521761 5014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:13.522068 5014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:13.522153 5014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:13.529558 5014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:09:13.529682 5014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:13.529721 5014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:13.529768 5014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:09:13.529814 5014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:09:13.539208 5014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35229
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:13.540561 5014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:13.542289 5014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:13.554693 5020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:13.554778 5019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:13.554719 5022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:13.555723 5014 server_base.cc:1034] running on GCE node
I20260927 22:09:13.556659 5014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:13.558300 5014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:13.559541 5014 hybrid_clock.cc:648] HybridClock initialized: now 1790546953559497 us; error 79 us; skew 500 ppm
I20260927 22:09:13.559912 5014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:09:13.562407 5014 webserver.cc:460] Webserver started at http://127.0.97.62:38213/ using document root <none> and password file <none>
I20260927 22:09:13.563218 5014 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:13.563362 5014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:13.563721 5014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:13.566967 5014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "3fe98f8308e54d14afc588bd361febe1"
format_stamp: "Formatted at 2026-09-27 22:09:13 on dist-test-slave-mp03"
server_key: "9f8531961d3aeb4c8a2b4b3b173a7511"
server_key_iv: "4d5c84471197740105b475eb5b86279a"
server_key_version: "encryptionkey@0"
I20260927 22:09:13.567850 5014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "3fe98f8308e54d14afc588bd361febe1"
format_stamp: "Formatted at 2026-09-27 22:09:13 on dist-test-slave-mp03"
server_key: "9f8531961d3aeb4c8a2b4b3b173a7511"
server_key_iv: "4d5c84471197740105b475eb5b86279a"
server_key_version: "encryptionkey@0"
I20260927 22:09:13.573555 5014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 22:09:13.577399 5028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:13.578930 5014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:09:13.579135 5014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "3fe98f8308e54d14afc588bd361febe1"
format_stamp: "Formatted at 2026-09-27 22:09:13 on dist-test-slave-mp03"
server_key: "9f8531961d3aeb4c8a2b4b3b173a7511"
server_key_iv: "4d5c84471197740105b475eb5b86279a"
server_key_version: "encryptionkey@0"
I20260927 22:09:13.579378 5014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:13.585760 5014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:13.586658 5014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:13.586970 5014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:13.605400 5014 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35229
I20260927 22:09:13.605422 5079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35229 every 8 connection(s)
I20260927 22:09:13.606942 5014 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:09:13.610695 5080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:13.616490 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 5014
I20260927 22:09:13.616719 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:09:13.617300 388 external_mini_cluster.cc:1428] Setting key b5af1bbc3710c166a00161113d105f3b
I20260927 22:09:13.622004 5080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fe98f8308e54d14afc588bd361febe1: Bootstrap starting.
I20260927 22:09:13.625712 5080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fe98f8308e54d14afc588bd361febe1: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:13.626541 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:13.619561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36652 (local address 127.0.97.62:35229)
0927 22:09:13.620597 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:13.620609 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:13.620652 (+ 43us) server_negotiation.cc:407] Connection header received
0927 22:09:13.622909 (+ 2257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:13.622934 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:13.623373 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:13.623730 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:13.624858 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:13.624864 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:13.624878 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:09:13.624892 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:13.624953 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:09:13.624969 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:13.625106 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:13.625253 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 22:09:13.625523 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":356,"threads_started":1}
I20260927 22:09:13.626928 5080 log.cc:824] T 00000000000000000000000000000000 P 3fe98f8308e54d14afc588bd361febe1: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:13.629906 5080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fe98f8308e54d14afc588bd361febe1: No bootstrap required, opened a new log
I20260927 22:09:13.637080 5080 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fe98f8308e54d14afc588bd361febe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe98f8308e54d14afc588bd361febe1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35229 } }
I20260927 22:09:13.637415 5080 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fe98f8308e54d14afc588bd361febe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:13.637485 5080 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fe98f8308e54d14afc588bd361febe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fe98f8308e54d14afc588bd361febe1, State: Initialized, Role: FOLLOWER
I20260927 22:09:13.637991 5080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fe98f8308e54d14afc588bd361febe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe98f8308e54d14afc588bd361febe1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35229 } }
I20260927 22:09:13.638208 5080 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fe98f8308e54d14afc588bd361febe1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:09:13.638353 5080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fe98f8308e54d14afc588bd361febe1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:09:13.638536 5080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fe98f8308e54d14afc588bd361febe1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:13.640722 5080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fe98f8308e54d14afc588bd361febe1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe98f8308e54d14afc588bd361febe1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35229 } }
I20260927 22:09:13.641216 5080 leader_election.cc:302] T 00000000000000000000000000000000 P 3fe98f8308e54d14afc588bd361febe1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fe98f8308e54d14afc588bd361febe1; no voters:
I20260927 22:09:13.641793 5080 leader_election.cc:288] T 00000000000000000000000000000000 P 3fe98f8308e54d14afc588bd361febe1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:09:13.641919 5085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fe98f8308e54d14afc588bd361febe1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:09:13.642707 5085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fe98f8308e54d14afc588bd361febe1 [term 1 LEADER]: Becoming Leader. State: Replica: 3fe98f8308e54d14afc588bd361febe1, State: Running, Role: LEADER
I20260927 22:09:13.643261 5085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fe98f8308e54d14afc588bd361febe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe98f8308e54d14afc588bd361febe1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35229 } }
I20260927 22:09:13.643934 5080 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fe98f8308e54d14afc588bd361febe1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:09:13.646494 5087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fe98f8308e54d14afc588bd361febe1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fe98f8308e54d14afc588bd361febe1. Latest consensus state: current_term: 1 leader_uuid: "3fe98f8308e54d14afc588bd361febe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe98f8308e54d14afc588bd361febe1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35229 } } }
I20260927 22:09:13.646960 5087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fe98f8308e54d14afc588bd361febe1 [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:13.647818 5086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fe98f8308e54d14afc588bd361febe1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fe98f8308e54d14afc588bd361febe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe98f8308e54d14afc588bd361febe1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35229 } } }
I20260927 22:09:13.648403 5086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fe98f8308e54d14afc588bd361febe1 [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:13.649804 5094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:09:13.654512 5094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:09:13.662985 5094 catalog_manager.cc:1347] Generated new cluster ID: 1a84953d72a84c449e65985115b69322
I20260927 22:09:13.663110 5094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:09:13.671046 5094 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:09:13.671468 5094 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:09:13.678269 5094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fe98f8308e54d14afc588bd361febe1: Generated new TSK 0
I20260927 22:09:13.679198 5094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:09:13.689147 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35229
--builtin_ntp_servers=127.0.97.20:45739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 22:09:14.016635 5104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:14.016911 5104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:14.017071 5104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:14.024268 5104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:14.024379 5104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:14.024572 5104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:09:14.034201 5104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:14.035576 5104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:14.037451 5104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:14.051100 5109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:14.051061 5110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:14.051023 5112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:14.051960 5104 server_base.cc:1034] running on GCE node
I20260927 22:09:14.052569 5104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:14.053558 5104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:14.054776 5104 hybrid_clock.cc:648] HybridClock initialized: now 1790546954054722 us; error 45 us; skew 500 ppm
I20260927 22:09:14.055099 5104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:09:14.057458 5104 webserver.cc:460] Webserver started at http://127.0.97.1:40737/ using document root <none> and password file <none>
I20260927 22:09:14.058149 5104 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:14.058290 5104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:14.058666 5104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:14.061568 5104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "4422f5d1a3a54cd28d6d84d5408f08ad"
format_stamp: "Formatted at 2026-09-27 22:09:14 on dist-test-slave-mp03"
server_key: "4cf3f6f4742976d344eb6ae1fd4e4c3f"
server_key_iv: "27bce0b48443cb35fdf3d9cb9b9b1c6f"
server_key_version: "encryptionkey@0"
I20260927 22:09:14.062342 5104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4422f5d1a3a54cd28d6d84d5408f08ad"
format_stamp: "Formatted at 2026-09-27 22:09:14 on dist-test-slave-mp03"
server_key: "4cf3f6f4742976d344eb6ae1fd4e4c3f"
server_key_iv: "27bce0b48443cb35fdf3d9cb9b9b1c6f"
server_key_version: "encryptionkey@0"
I20260927 22:09:14.067349 5104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.002s
I20260927 22:09:14.071369 5118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:14.073009 5104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:09:14.073171 5104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "4422f5d1a3a54cd28d6d84d5408f08ad"
format_stamp: "Formatted at 2026-09-27 22:09:14 on dist-test-slave-mp03"
server_key: "4cf3f6f4742976d344eb6ae1fd4e4c3f"
server_key_iv: "27bce0b48443cb35fdf3d9cb9b9b1c6f"
server_key_version: "encryptionkey@0"
I20260927 22:09:14.073411 5104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:14.078833 5104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:14.079582 5104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:14.079849 5104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:14.080751 5104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:14.082036 5104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:14.082154 5104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:14.082285 5104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:14.082370 5104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:14.115252 5104 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44563
I20260927 22:09:14.115302 5230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44563 every 8 connection(s)
I20260927 22:09:14.116562 5104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:09:14.118099 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 5104
I20260927 22:09:14.118288 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:09:14.119093 388 external_mini_cluster.cc:1428] Setting key 66d9dcde5e035cf96ec140cbd7646615
I20260927 22:09:14.122200 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35229
--builtin_ntp_servers=127.0.97.20:45739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 22:09:14.126705 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:14.119682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55865 (local address 127.0.97.62:35229)
0927 22:09:14.119910 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:14.119916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:14.121506 (+ 1590us) server_negotiation.cc:407] Connection header received
0927 22:09:14.124330 (+ 2824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:14.124333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:14.124382 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:14.124459 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:14.125907 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:14.125915 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:14.125922 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:09:14.125941 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:14.125979 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:09:14.125994 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:14.126109 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:14.126321 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 22:09:14.126379 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:09:14.127882 5233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:14.120229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35229 (local address 127.0.97.1:55865)
0927 22:09:14.121333 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:14.121393 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:14.124018 (+ 2625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:14.124726 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:14.124740 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:14.124868 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:14.125560 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:09:14.125566 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:14.125619 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:09:14.125731 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:14.126103 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:14.126115 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:14.126125 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 22:09:14.126394 (+ 269us) client_negotiation.cc:240] Negotiation successful
0927 22:09:14.126670 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":287,"threads_started":1}
I20260927 22:09:14.129508 5231 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35229
I20260927 22:09:14.129793 5231 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:14.130316 5231 heartbeater.cc:511] Master 127.0.97.62:35229 requested a full tablet report, sending...
I20260927 22:09:14.132478 5045 ts_manager.cc:194] Registered new tserver with Master: 4422f5d1a3a54cd28d6d84d5408f08ad (127.0.97.1:44563)
W20260927 22:09:14.451794 5234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:14.452086 5234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:14.452186 5234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:14.459553 5234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:14.459678 5234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:14.459798 5234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:09:14.469204 5234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:14.470433 5234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:14.472267 5234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:14.485764 5239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:14.485886 5240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:14.485780 5242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:14.486671 5234 server_base.cc:1034] running on GCE node
I20260927 22:09:14.487286 5234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:14.488225 5234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:14.489471 5234 hybrid_clock.cc:648] HybridClock initialized: now 1790546954489419 us; error 43 us; skew 500 ppm
I20260927 22:09:14.489821 5234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:09:14.492106 5234 webserver.cc:460] Webserver started at http://127.0.97.2:39151/ using document root <none> and password file <none>
I20260927 22:09:14.492816 5234 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:14.492972 5234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:14.493294 5234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:14.496228 5234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "f8c177d4425e4b09a6059a46558a5f11"
format_stamp: "Formatted at 2026-09-27 22:09:14 on dist-test-slave-mp03"
server_key: "45306e1e103fafb4bf44194b3965df0f"
server_key_iv: "a1ee5b4db78a5506189e308295a35129"
server_key_version: "encryptionkey@0"
I20260927 22:09:14.496999 5234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f8c177d4425e4b09a6059a46558a5f11"
format_stamp: "Formatted at 2026-09-27 22:09:14 on dist-test-slave-mp03"
server_key: "45306e1e103fafb4bf44194b3965df0f"
server_key_iv: "a1ee5b4db78a5506189e308295a35129"
server_key_version: "encryptionkey@0"
I20260927 22:09:14.501976 5234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:09:14.505393 5248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:14.506750 5234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:09:14.506877 5234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "f8c177d4425e4b09a6059a46558a5f11"
format_stamp: "Formatted at 2026-09-27 22:09:14 on dist-test-slave-mp03"
server_key: "45306e1e103fafb4bf44194b3965df0f"
server_key_iv: "a1ee5b4db78a5506189e308295a35129"
server_key_version: "encryptionkey@0"
I20260927 22:09:14.507054 5234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:14.512422 5234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:14.513134 5234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:14.513418 5234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:14.514261 5234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:14.515614 5234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:14.515698 5234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:14.515786 5234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:14.515839 5234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:14.547508 5234 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41877
I20260927 22:09:14.547533 5360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41877 every 8 connection(s)
I20260927 22:09:14.548799 5234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:09:14.554350 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 5234
I20260927 22:09:14.554540 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:09:14.555038 388 external_mini_cluster.cc:1428] Setting key 6f1a44343a15859e956e3361134ff525
I20260927 22:09:14.557790 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:14.551805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42779 (local address 127.0.97.62:35229)
0927 22:09:14.552033 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:14.552040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:14.553519 (+ 1479us) server_negotiation.cc:407] Connection header received
0927 22:09:14.555322 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:14.555328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:14.555401 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:14.555534 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:14.557064 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:14.557068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:14.557071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:09:14.557086 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:14.557142 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:09:14.557157 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:14.557287 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:14.557455 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 22:09:14.557520 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:09:14.558305 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35229
--builtin_ntp_servers=127.0.97.20:45739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 22:09:14.558905 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:14.552444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35229 (local address 127.0.97.2:42779)
0927 22:09:14.553346 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:14.553382 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:14.555007 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:14.555787 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:14.555799 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:14.555945 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:14.556696 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:09:14.556699 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:14.556764 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:09:14.556894 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:14.557263 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:14.557271 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:14.557277 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 22:09:14.557496 (+ 219us) client_negotiation.cc:240] Negotiation successful
0927 22:09:14.557764 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":273,"threads_started":1}
I20260927 22:09:14.560416 5361 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35229
I20260927 22:09:14.560740 5361 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:14.561385 5361 heartbeater.cc:511] Master 127.0.97.62:35229 requested a full tablet report, sending...
I20260927 22:09:14.562894 5045 ts_manager.cc:194] Registered new tserver with Master: f8c177d4425e4b09a6059a46558a5f11 (127.0.97.2:41877)
W20260927 22:09:14.897056 5364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:14.897365 5364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:14.897549 5364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:14.905084 5364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:14.905211 5364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:14.905428 5364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:09:14.915031 5364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:14.916401 5364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:14.918229 5364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:14.932282 5369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:14.932229 5372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:14.932332 5370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:14.936193 5364 server_base.cc:1034] running on GCE node
I20260927 22:09:14.936851 5364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:14.937898 5364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:14.939117 5364 hybrid_clock.cc:648] HybridClock initialized: now 1790546954939049 us; error 68 us; skew 500 ppm
I20260927 22:09:14.939441 5364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:09:14.949672 5364 webserver.cc:460] Webserver started at http://127.0.97.3:43425/ using document root <none> and password file <none>
I20260927 22:09:14.950446 5364 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:14.950626 5364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:14.951015 5364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:14.954070 5364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "8ba5555424504c538512e9db4dd03a95"
format_stamp: "Formatted at 2026-09-27 22:09:14 on dist-test-slave-mp03"
server_key: "6498170399d34320d35b09909c0c2576"
server_key_iv: "cbe069c6f4a19c91d0a399e0ae4cbf67"
server_key_version: "encryptionkey@0"
I20260927 22:09:14.954963 5364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8ba5555424504c538512e9db4dd03a95"
format_stamp: "Formatted at 2026-09-27 22:09:14 on dist-test-slave-mp03"
server_key: "6498170399d34320d35b09909c0c2576"
server_key_iv: "cbe069c6f4a19c91d0a399e0ae4cbf67"
server_key_version: "encryptionkey@0"
I20260927 22:09:14.960235 5364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:09:14.963693 5378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:14.965058 5364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:09:14.965229 5364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "8ba5555424504c538512e9db4dd03a95"
format_stamp: "Formatted at 2026-09-27 22:09:14 on dist-test-slave-mp03"
server_key: "6498170399d34320d35b09909c0c2576"
server_key_iv: "cbe069c6f4a19c91d0a399e0ae4cbf67"
server_key_version: "encryptionkey@0"
I20260927 22:09:14.965469 5364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:14.971158 5364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:14.971946 5364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:14.972203 5364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:14.973098 5364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:14.974445 5364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:14.974562 5364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:14.974772 5364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:14.974864 5364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:15.017601 5364 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45835
I20260927 22:09:15.017627 5490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45835 every 8 connection(s)
I20260927 22:09:15.019377 5364 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:09:15.029570 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 5364
I20260927 22:09:15.029750 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:09:15.030105 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:15.022470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52287 (local address 127.0.97.62:35229)
0927 22:09:15.022734 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:15.022742 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:15.024515 (+ 1773us) server_negotiation.cc:407] Connection header received
0927 22:09:15.027323 (+ 2808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:15.027329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:15.027417 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:15.027542 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:15.029217 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:15.029225 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:15.029231 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:09:15.029252 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:15.029290 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:09:15.029306 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:15.029469 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:15.029780 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 22:09:15.029847 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 22:09:15.030285 388 external_mini_cluster.cc:1428] Setting key 4eb23d29b3f9690af97123bab6260f5c
I20260927 22:09:15.031916 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:15.023121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35229 (local address 127.0.97.3:52287)
0927 22:09:15.024329 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:15.024383 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:15.027019 (+ 2636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:15.027835 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:15.027850 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:15.027989 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:15.028825 (+ 836us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:09:15.028829 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:15.028887 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:09:15.029025 (+ 138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:15.029525 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:15.029538 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:15.029549 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 22:09:15.030392 (+ 843us) client_negotiation.cc:240] Negotiation successful
0927 22:09:15.030721 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":348,"threads_started":1}
I20260927 22:09:15.034086 5491 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35229
I20260927 22:09:15.034329 5491 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:15.034922 5491 heartbeater.cc:511] Master 127.0.97.62:35229 requested a full tablet report, sending...
I20260927 22:09:15.036520 5045 ts_manager.cc:194] Registered new tserver with Master: 8ba5555424504c538512e9db4dd03a95 (127.0.97.3:45835)
I20260927 22:09:15.045883 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:09:15.053103 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:15.050465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36660 (local address 127.0.97.62:35229)
0927 22:09:15.050693 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:15.050702 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:15.051205 (+ 503us) server_negotiation.cc:407] Connection header received
0927 22:09:15.051422 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:15.051433 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:15.051486 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:15.051567 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:15.052425 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:15.052429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:15.052432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:09:15.052446 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:15.052475 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:09:15.052486 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:15.052580 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:15.052736 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 22:09:15.052780 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:09:15.057279 5045 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:09:15.063985 5045 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:09:15.096357 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:15.077544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56680 (local address 127.0.97.1:44563)
0927 22:09:15.092349 (+ 14805us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:15.092362 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:15.092402 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:09:15.092492 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:15.092505 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:15.093151 (+ 646us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:15.093344 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:15.095432 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:15.095440 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:15.095455 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:09:15.095478 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:15.095531 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:09:15.095551 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:15.095671 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:15.095877 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 22:09:15.095971 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14575,"thread_start_us":15233,"threads_started":1}
I20260927 22:09:15.096431 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:15.076819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44563 (local address 127.0.0.1:56680)
0927 22:09:15.077634 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:15.077691 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:15.077842 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:15.093354 (+ 15512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:15.093363 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:15.093468 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:15.094746 (+ 1278us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:09:15.094749 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:15.094783 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:09:15.094871 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:15.095639 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:15.095653 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:15.095665 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 22:09:15.095883 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 22:09:15.095987 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":918,"threads_started":1}
I20260927 22:09:15.098802 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:15.080148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59516 (local address 127.0.97.3:45835)
0927 22:09:15.096364 (+ 16216us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:15.096373 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:15.096408 (+ 35us) server_negotiation.cc:407] Connection header received
0927 22:09:15.096473 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:15.096481 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:15.096689 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:15.096834 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:15.097982 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:15.097988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:15.097998 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:09:15.098012 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:15.098052 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:09:15.098065 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:15.098236 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:15.098296 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 22:09:15.098380 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16037,"thread_start_us":17607,"threads_started":1}
I20260927 22:09:15.098810 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:15.079811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45835 (local address 127.0.0.1:59516)
0927 22:09:15.094579 (+ 14768us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:15.094621 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:15.094714 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:15.096833 (+ 2119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:15.096838 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:15.096860 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:15.097775 (+ 915us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:09:15.097777 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:15.097789 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:09:15.097819 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:15.098175 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:15.098182 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:15.098189 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:09:15.098308 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 22:09:15.098355 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14647,"thread_start_us":7417,"threads_started":1}
I20260927 22:09:15.099407 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:15.077011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41877 (local address 127.0.0.1:58814)
0927 22:09:15.087562 (+ 10551us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:15.087588 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:15.087705 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:15.096862 (+ 9157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:15.096866 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:15.096890 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:15.098029 (+ 1139us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:09:15.098032 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:15.098049 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:09:15.098088 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:15.098551 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:15.098555 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:15.098563 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:09:15.098949 (+ 386us) client_negotiation.cc:240] Negotiation successful
0927 22:09:15.098993 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10350,"thread_start_us":17242,"threads_started":1}
I20260927 22:09:15.100247 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:15.077346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58814 (local address 127.0.97.2:41877)
0927 22:09:15.095170 (+ 17824us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:15.095178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:15.095204 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:09:15.095275 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:15.095283 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:15.095515 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:15.095659 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:15.098251 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:15.098258 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:15.098270 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:09:15.098297 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:15.098349 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:09:15.098373 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:15.098497 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:15.099859 (+ 1362us) server_negotiation.cc:299] Negotiation successful
0927 22:09:15.099950 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17673,"thread_start_us":17549,"threads_started":1}
I20260927 22:09:15.100533 5166 tablet_service.cc:1467] Processing CreateTablet for tablet 13ef2ee2866a4b78b9b1feb433143f5f (DEFAULT_TABLE table=test-table [id=41e95bd59bc64546836f0dad8f59d5e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:09:15.101841 5166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ef2ee2866a4b78b9b1feb433143f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:15.102556 5426 tablet_service.cc:1467] Processing CreateTablet for tablet 13ef2ee2866a4b78b9b1feb433143f5f (DEFAULT_TABLE table=test-table [id=41e95bd59bc64546836f0dad8f59d5e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:09:15.103956 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ef2ee2866a4b78b9b1feb433143f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:15.103504 5296 tablet_service.cc:1467] Processing CreateTablet for tablet 13ef2ee2866a4b78b9b1feb433143f5f (DEFAULT_TABLE table=test-table [id=41e95bd59bc64546836f0dad8f59d5e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:09:15.104718 5296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ef2ee2866a4b78b9b1feb433143f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:15.112545 5509 tablet_bootstrap.cc:492] T 13ef2ee2866a4b78b9b1feb433143f5f P 4422f5d1a3a54cd28d6d84d5408f08ad: Bootstrap starting.
I20260927 22:09:15.116197 5509 tablet_bootstrap.cc:654] T 13ef2ee2866a4b78b9b1feb433143f5f P 4422f5d1a3a54cd28d6d84d5408f08ad: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:15.117591 5509 log.cc:824] T 13ef2ee2866a4b78b9b1feb433143f5f P 4422f5d1a3a54cd28d6d84d5408f08ad: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:15.126510 5512 tablet_bootstrap.cc:492] T 13ef2ee2866a4b78b9b1feb433143f5f P f8c177d4425e4b09a6059a46558a5f11: Bootstrap starting.
I20260927 22:09:15.128252 5509 tablet_bootstrap.cc:492] T 13ef2ee2866a4b78b9b1feb433143f5f P 4422f5d1a3a54cd28d6d84d5408f08ad: No bootstrap required, opened a new log
I20260927 22:09:15.128536 5509 ts_tablet_manager.cc:1397] T 13ef2ee2866a4b78b9b1feb433143f5f P 4422f5d1a3a54cd28d6d84d5408f08ad: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20260927 22:09:15.130170 5512 tablet_bootstrap.cc:654] T 13ef2ee2866a4b78b9b1feb433143f5f P f8c177d4425e4b09a6059a46558a5f11: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:15.131585 5512 log.cc:824] T 13ef2ee2866a4b78b9b1feb433143f5f P f8c177d4425e4b09a6059a46558a5f11: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:15.132174 5510 tablet_bootstrap.cc:492] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95: Bootstrap starting.
I20260927 22:09:15.135116 5512 tablet_bootstrap.cc:492] T 13ef2ee2866a4b78b9b1feb433143f5f P f8c177d4425e4b09a6059a46558a5f11: No bootstrap required, opened a new log
I20260927 22:09:15.135393 5512 ts_tablet_manager.cc:1397] T 13ef2ee2866a4b78b9b1feb433143f5f P f8c177d4425e4b09a6059a46558a5f11: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260927 22:09:15.135476 5231 heartbeater.cc:503] Master 127.0.97.62:35229 was elected leader, sending a full tablet report...
I20260927 22:09:15.136538 5510 tablet_bootstrap.cc:654] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:15.137679 5510 log.cc:824] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:15.140354 5509 raft_consensus.cc:348] T 13ef2ee2866a4b78b9b1feb433143f5f P 4422f5d1a3a54cd28d6d84d5408f08ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4422f5d1a3a54cd28d6d84d5408f08ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44563 } } peers { permanent_uuid: "f8c177d4425e4b09a6059a46558a5f11" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41877 } } peers { permanent_uuid: "8ba5555424504c538512e9db4dd03a95" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45835 } }
I20260927 22:09:15.140719 5509 raft_consensus.cc:374] T 13ef2ee2866a4b78b9b1feb433143f5f P 4422f5d1a3a54cd28d6d84d5408f08ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:15.140815 5509 raft_consensus.cc:729] T 13ef2ee2866a4b78b9b1feb433143f5f P 4422f5d1a3a54cd28d6d84d5408f08ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4422f5d1a3a54cd28d6d84d5408f08ad, State: Initialized, Role: FOLLOWER
I20260927 22:09:15.141424 5509 consensus_queue.cc:260] T 13ef2ee2866a4b78b9b1feb433143f5f P 4422f5d1a3a54cd28d6d84d5408f08ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4422f5d1a3a54cd28d6d84d5408f08ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44563 } } peers { permanent_uuid: "f8c177d4425e4b09a6059a46558a5f11" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41877 } } peers { permanent_uuid: "8ba5555424504c538512e9db4dd03a95" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45835 } }
I20260927 22:09:15.142743 5510 tablet_bootstrap.cc:492] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95: No bootstrap required, opened a new log
I20260927 22:09:15.143064 5510 ts_tablet_manager.cc:1397] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260927 22:09:15.144258 5512 raft_consensus.cc:348] T 13ef2ee2866a4b78b9b1feb433143f5f P f8c177d4425e4b09a6059a46558a5f11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4422f5d1a3a54cd28d6d84d5408f08ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44563 } } peers { permanent_uuid: "f8c177d4425e4b09a6059a46558a5f11" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41877 } } peers { permanent_uuid: "8ba5555424504c538512e9db4dd03a95" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45835 } }
I20260927 22:09:15.144587 5512 raft_consensus.cc:374] T 13ef2ee2866a4b78b9b1feb433143f5f P f8c177d4425e4b09a6059a46558a5f11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:15.144701 5512 raft_consensus.cc:729] T 13ef2ee2866a4b78b9b1feb433143f5f P f8c177d4425e4b09a6059a46558a5f11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c177d4425e4b09a6059a46558a5f11, State: Initialized, Role: FOLLOWER
I20260927 22:09:15.145324 5512 consensus_queue.cc:260] T 13ef2ee2866a4b78b9b1feb433143f5f P f8c177d4425e4b09a6059a46558a5f11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4422f5d1a3a54cd28d6d84d5408f08ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44563 } } peers { permanent_uuid: "f8c177d4425e4b09a6059a46558a5f11" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41877 } } peers { permanent_uuid: "8ba5555424504c538512e9db4dd03a95" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45835 } }
I20260927 22:09:15.149410 5509 ts_tablet_manager.cc:1428] T 13ef2ee2866a4b78b9b1feb433143f5f P 4422f5d1a3a54cd28d6d84d5408f08ad: Time spent starting tablet: real 0.021s user 0.016s sys 0.005s
I20260927 22:09:15.150344 5361 heartbeater.cc:503] Master 127.0.97.62:35229 was elected leader, sending a full tablet report...
I20260927 22:09:15.150961 5512 ts_tablet_manager.cc:1428] T 13ef2ee2866a4b78b9b1feb433143f5f P f8c177d4425e4b09a6059a46558a5f11: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260927 22:09:15.154500 5510 raft_consensus.cc:348] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4422f5d1a3a54cd28d6d84d5408f08ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44563 } } peers { permanent_uuid: "f8c177d4425e4b09a6059a46558a5f11" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41877 } } peers { permanent_uuid: "8ba5555424504c538512e9db4dd03a95" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45835 } }
I20260927 22:09:15.154964 5510 raft_consensus.cc:374] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:15.155040 5510 raft_consensus.cc:729] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ba5555424504c538512e9db4dd03a95, State: Initialized, Role: FOLLOWER
I20260927 22:09:15.155525 5510 consensus_queue.cc:260] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4422f5d1a3a54cd28d6d84d5408f08ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44563 } } peers { permanent_uuid: "f8c177d4425e4b09a6059a46558a5f11" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41877 } } peers { permanent_uuid: "8ba5555424504c538512e9db4dd03a95" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45835 } }
I20260927 22:09:15.156406 5491 heartbeater.cc:503] Master 127.0.97.62:35229 was elected leader, sending a full tablet report...
I20260927 22:09:15.157048 5510 ts_tablet_manager.cc:1428] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260927 22:09:15.271179 5492 tablet.cc:2369] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:09:15.301075 5362 tablet.cc:2369] T 13ef2ee2866a4b78b9b1feb433143f5f P f8c177d4425e4b09a6059a46558a5f11: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:09:15.369462 5232 tablet.cc:2369] T 13ef2ee2866a4b78b9b1feb433143f5f P 4422f5d1a3a54cd28d6d84d5408f08ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:09:15.456627 5517 raft_consensus.cc:482] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:09:15.456964 5517 raft_consensus.cc:504] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4422f5d1a3a54cd28d6d84d5408f08ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44563 } } peers { permanent_uuid: "f8c177d4425e4b09a6059a46558a5f11" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41877 } } peers { permanent_uuid: "8ba5555424504c538512e9db4dd03a95" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45835 } }
I20260927 22:09:15.458737 5517 leader_election.cc:288] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4422f5d1a3a54cd28d6d84d5408f08ad (127.0.97.1:44563), f8c177d4425e4b09a6059a46558a5f11 (127.0.97.2:41877)
I20260927 22:09:15.461248 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:15.459075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52813 (local address 127.0.97.1:44563)
0927 22:09:15.459287 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:15.459295 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:15.459348 (+ 53us) server_negotiation.cc:407] Connection header received
0927 22:09:15.459547 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:15.459552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:15.459617 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:15.459737 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:15.460659 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:15.460665 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:15.460669 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:09:15.460687 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:15.460723 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:09:15.460734 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:15.460848 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:15.460944 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 22:09:15.460998 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:09:15.461256 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:15.459062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44563 (local address 127.0.97.3:52813)
0927 22:09:15.459252 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:15.459275 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:15.459370 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:15.459780 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:15.459784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:15.459806 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:15.460455 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:09:15.460458 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:15.460471 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:09:15.460513 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:15.460823 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:15.460827 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:15.460832 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:09:15.460949 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 22:09:15.460998 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 22:09:15.461954 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:15.459387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57137 (local address 127.0.97.2:41877)
0927 22:09:15.459573 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:15.459579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:15.459998 (+ 419us) server_negotiation.cc:407] Connection header received
0927 22:09:15.460172 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:15.460177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:15.460237 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:15.460354 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:15.461286 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:15.461291 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:15.461296 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:09:15.461317 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:15.461359 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:09:15.461378 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:15.461483 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:15.461621 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 22:09:15.461677 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 22:09:15.462097 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:15.459134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41877 (local address 127.0.97.3:57137)
0927 22:09:15.459894 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:15.459908 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:15.460036 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:15.460395 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:15.460404 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:15.460425 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:15.461092 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:09:15.461095 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:15.461106 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:09:15.461139 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:15.461501 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:15.461506 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:15.461511 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:09:15.461621 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 22:09:15.461665 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":390,"threads_started":1}
I20260927 22:09:15.462289 5186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ef2ee2866a4b78b9b1feb433143f5f" candidate_uuid: "8ba5555424504c538512e9db4dd03a95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4422f5d1a3a54cd28d6d84d5408f08ad" is_pre_election: true
I20260927 22:09:15.462970 5186 raft_consensus.cc:2454] T 13ef2ee2866a4b78b9b1feb433143f5f P 4422f5d1a3a54cd28d6d84d5408f08ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ba5555424504c538512e9db4dd03a95 in term 0.
I20260927 22:09:15.463052 5316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ef2ee2866a4b78b9b1feb433143f5f" candidate_uuid: "8ba5555424504c538512e9db4dd03a95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c177d4425e4b09a6059a46558a5f11" is_pre_election: true
I20260927 22:09:15.463636 5316 raft_consensus.cc:2454] T 13ef2ee2866a4b78b9b1feb433143f5f P f8c177d4425e4b09a6059a46558a5f11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ba5555424504c538512e9db4dd03a95 in term 0.
I20260927 22:09:15.463797 5382 leader_election.cc:302] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4422f5d1a3a54cd28d6d84d5408f08ad, 8ba5555424504c538512e9db4dd03a95; no voters:
I20260927 22:09:15.464213 5517 raft_consensus.cc:2781] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:09:15.464386 5517 raft_consensus.cc:482] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:09:15.464493 5517 raft_consensus.cc:3037] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:15.466727 5517 raft_consensus.cc:504] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4422f5d1a3a54cd28d6d84d5408f08ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44563 } } peers { permanent_uuid: "f8c177d4425e4b09a6059a46558a5f11" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41877 } } peers { permanent_uuid: "8ba5555424504c538512e9db4dd03a95" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45835 } }
I20260927 22:09:15.467319 5517 leader_election.cc:288] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95 [CANDIDATE]: Term 1 election: Requested vote from peers 4422f5d1a3a54cd28d6d84d5408f08ad (127.0.97.1:44563), f8c177d4425e4b09a6059a46558a5f11 (127.0.97.2:41877)
I20260927 22:09:15.467902 5186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ef2ee2866a4b78b9b1feb433143f5f" candidate_uuid: "8ba5555424504c538512e9db4dd03a95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4422f5d1a3a54cd28d6d84d5408f08ad"
I20260927 22:09:15.467902 5316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ef2ee2866a4b78b9b1feb433143f5f" candidate_uuid: "8ba5555424504c538512e9db4dd03a95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c177d4425e4b09a6059a46558a5f11"
I20260927 22:09:15.468142 5186 raft_consensus.cc:3037] T 13ef2ee2866a4b78b9b1feb433143f5f P 4422f5d1a3a54cd28d6d84d5408f08ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:15.468142 5316 raft_consensus.cc:3037] T 13ef2ee2866a4b78b9b1feb433143f5f P f8c177d4425e4b09a6059a46558a5f11 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:15.471838 5186 raft_consensus.cc:2454] T 13ef2ee2866a4b78b9b1feb433143f5f P 4422f5d1a3a54cd28d6d84d5408f08ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ba5555424504c538512e9db4dd03a95 in term 1.
I20260927 22:09:15.471838 5316 raft_consensus.cc:2454] T 13ef2ee2866a4b78b9b1feb433143f5f P f8c177d4425e4b09a6059a46558a5f11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ba5555424504c538512e9db4dd03a95 in term 1.
I20260927 22:09:15.472397 5382 leader_election.cc:302] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4422f5d1a3a54cd28d6d84d5408f08ad, 8ba5555424504c538512e9db4dd03a95, f8c177d4425e4b09a6059a46558a5f11; no voters:
I20260927 22:09:15.472833 5517 raft_consensus.cc:2781] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:09:15.473263 5517 raft_consensus.cc:686] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95 [term 1 LEADER]: Becoming Leader. State: Replica: 8ba5555424504c538512e9db4dd03a95, State: Running, Role: LEADER
I20260927 22:09:15.473809 5517 consensus_queue.cc:237] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4422f5d1a3a54cd28d6d84d5408f08ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44563 } } peers { permanent_uuid: "f8c177d4425e4b09a6059a46558a5f11" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41877 } } peers { permanent_uuid: "8ba5555424504c538512e9db4dd03a95" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45835 } }
I20260927 22:09:15.478505 5044 catalog_manager.cc:5495] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ba5555424504c538512e9db4dd03a95 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "8ba5555424504c538512e9db4dd03a95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4422f5d1a3a54cd28d6d84d5408f08ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c177d4425e4b09a6059a46558a5f11" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ba5555424504c538512e9db4dd03a95" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45835 } health_report { overall_health: HEALTHY } } }
I20260927 22:09:15.544970 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:15.542893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.0.97.3:45835)
0927 22:09:15.543065 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:15.543070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:15.543087 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:09:15.543197 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:15.543201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:15.543273 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:15.543353 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:15.544404 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:15.544407 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:15.544410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:09:15.544434 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:15.544461 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:09:15.544472 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:15.544605 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:15.544648 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 22:09:15.544718 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:09:15.553884 5186 raft_consensus.cc:1261] T 13ef2ee2866a4b78b9b1feb433143f5f P 4422f5d1a3a54cd28d6d84d5408f08ad [term 1 FOLLOWER]: Refusing update from remote peer 8ba5555424504c538512e9db4dd03a95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:09:15.553894 5316 raft_consensus.cc:1261] T 13ef2ee2866a4b78b9b1feb433143f5f P f8c177d4425e4b09a6059a46558a5f11 [term 1 FOLLOWER]: Refusing update from remote peer 8ba5555424504c538512e9db4dd03a95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:09:15.554940 5519 consensus_queue.cc:1035] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4422f5d1a3a54cd28d6d84d5408f08ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:09:15.555356 5517 consensus_queue.cc:1035] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c177d4425e4b09a6059a46558a5f11" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:09:15.587129 5045 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:09:15.587350 5045 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:09:15.590826 5045 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3fe98f8308e54d14afc588bd361febe1: Sending DeleteTablet for 3 replicas of tablet 13ef2ee2866a4b78b9b1feb433143f5f
I20260927 22:09:15.591988 5166 tablet_service.cc:1514] Processing DeleteTablet for tablet 13ef2ee2866a4b78b9b1feb433143f5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:15 UTC) from {username='slave'} at 127.0.0.1:56680
I20260927 22:09:15.592115 5296 tablet_service.cc:1514] Processing DeleteTablet for tablet 13ef2ee2866a4b78b9b1feb433143f5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:15 UTC) from {username='slave'} at 127.0.0.1:58814
I20260927 22:09:15.592336 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet 13ef2ee2866a4b78b9b1feb433143f5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:15 UTC) from {username='slave'} at 127.0.0.1:59516
I20260927 22:09:15.594707 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 5104
I20260927 22:09:15.606267 5534 tablet_replica.cc:330] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95: stopping tablet replica
I20260927 22:09:15.606980 5534 raft_consensus.cc:2229] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:09:15.607841 5534 raft_consensus.cc:2258] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:09:15.607872 5532 tablet_replica.cc:330] T 13ef2ee2866a4b78b9b1feb433143f5f P 4422f5d1a3a54cd28d6d84d5408f08ad: stopping tablet replica
I20260927 22:09:15.608892 5532 raft_consensus.cc:2229] T 13ef2ee2866a4b78b9b1feb433143f5f P 4422f5d1a3a54cd28d6d84d5408f08ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:09:15.609378 5532 raft_consensus.cc:2258] T 13ef2ee2866a4b78b9b1feb433143f5f P 4422f5d1a3a54cd28d6d84d5408f08ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:09:15.611160 5534 ts_tablet_manager.cc:1905] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:15.613706 5532 ts_tablet_manager.cc:1905] T 13ef2ee2866a4b78b9b1feb433143f5f P 4422f5d1a3a54cd28d6d84d5408f08ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:15.618628 5534 ts_tablet_manager.cc:1918] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:09:15.618816 5534 log.cc:1196] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/13ef2ee2866a4b78b9b1feb433143f5f
I20260927 22:09:15.619354 5534 ts_tablet_manager.cc:1939] T 13ef2ee2866a4b78b9b1feb433143f5f P 8ba5555424504c538512e9db4dd03a95: Deleting consensus metadata
I20260927 22:09:15.621330 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:15.596177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56692 (local address 127.0.97.1:44563)
0927 22:09:15.596352 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:15.596359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:15.612046 (+ 15687us) server_negotiation.cc:407] Connection header received
0927 22:09:15.612258 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:15.612266 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:15.612376 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:15.612534 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:15.620220 (+ 7686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:15.620226 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:15.620230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:09:15.620276 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:15.620326 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:09:15.620349 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:15.620501 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:15.620836 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 22:09:15.620919 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:09:15.621290 5029 catalog_manager.cc:4841] TS 8ba5555424504c538512e9db4dd03a95 (127.0.97.3:45835): tablet 13ef2ee2866a4b78b9b1feb433143f5f (table test-table [id=41e95bd59bc64546836f0dad8f59d5e5]) successfully deleted
I20260927 22:09:15.622143 5533 tablet_replica.cc:330] T 13ef2ee2866a4b78b9b1feb433143f5f P f8c177d4425e4b09a6059a46558a5f11: stopping tablet replica
I20260927 22:09:15.622663 5226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:15.622624 5533 raft_consensus.cc:2229] T 13ef2ee2866a4b78b9b1feb433143f5f P f8c177d4425e4b09a6059a46558a5f11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:09:15.623109 5533 raft_consensus.cc:2258] T 13ef2ee2866a4b78b9b1feb433143f5f P f8c177d4425e4b09a6059a46558a5f11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:09:15.625290 5533 ts_tablet_manager.cc:1905] T 13ef2ee2866a4b78b9b1feb433143f5f P f8c177d4425e4b09a6059a46558a5f11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:15.632045 5533 ts_tablet_manager.cc:1918] T 13ef2ee2866a4b78b9b1feb433143f5f P f8c177d4425e4b09a6059a46558a5f11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:09:15.632210 5533 log.cc:1196] T 13ef2ee2866a4b78b9b1feb433143f5f P f8c177d4425e4b09a6059a46558a5f11: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/13ef2ee2866a4b78b9b1feb433143f5f
I20260927 22:09:15.632723 5533 ts_tablet_manager.cc:1939] T 13ef2ee2866a4b78b9b1feb433143f5f P f8c177d4425e4b09a6059a46558a5f11: Deleting consensus metadata
I20260927 22:09:15.634624 5030 catalog_manager.cc:4841] TS f8c177d4425e4b09a6059a46558a5f11 (127.0.97.2:41877): tablet 13ef2ee2866a4b78b9b1feb433143f5f (table test-table [id=41e95bd59bc64546836f0dad8f59d5e5]) successfully deleted
I20260927 22:09:15.725493 5532 ts_tablet_manager.cc:1918] T 13ef2ee2866a4b78b9b1feb433143f5f P 4422f5d1a3a54cd28d6d84d5408f08ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:09:15.725746 5532 log.cc:1196] T 13ef2ee2866a4b78b9b1feb433143f5f P 4422f5d1a3a54cd28d6d84d5408f08ad: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546912983837-388-0/minicluster-data/ts-0/wal/wals/13ef2ee2866a4b78b9b1feb433143f5f
I20260927 22:09:15.726354 5532 ts_tablet_manager.cc:1939] T 13ef2ee2866a4b78b9b1feb433143f5f P 4422f5d1a3a54cd28d6d84d5408f08ad: Deleting consensus metadata
I20260927 22:09:15.726956 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 5104
I20260927 22:09:15.743427 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 5234
W20260927 22:09:15.743523 5032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44563 (error 108)
I20260927 22:09:15.746703 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:15.745884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44563 (local address 127.0.0.1:56700)
0927 22:09:15.746318 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:15.746452 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":234,"threads_started":1}
I20260927 22:09:15.746764 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:15.744511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58816 (local address 127.0.97.2:41877)
0927 22:09:15.744706 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:15.744712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:15.744737 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:09:15.744810 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:15.744816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:15.744918 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:15.745028 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:15.746150 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:15.746156 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:15.746161 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:09:15.746180 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:15.746217 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:09:15.746234 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:15.746352 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:15.746448 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 22:09:15.746508 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 22:09:15.746955 5032 catalog_manager.cc:4584] TS 4422f5d1a3a54cd28d6d84d5408f08ad (127.0.97.1:44563): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 13ef2ee2866a4b78b9b1feb433143f5f: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44563: connect: Connection refused (error 111)
I20260927 22:09:15.747444 5356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:15.800776 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:15.800261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44563 (local address 127.0.0.1:56702)
0927 22:09:15.800499 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:15.800599 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 22:09:15.801964 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:15.801520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44563 (local address 127.0.0.1:56710)
0927 22:09:15.801690 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:15.801788 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 22:09:15.836181 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 5234
I20260927 22:09:15.859977 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 5364
I20260927 22:09:15.862313 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:15.861932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44563 (local address 127.0.0.1:56718)
0927 22:09:15.862100 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:15.862181 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 22:09:15.863555 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:15.863109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44563 (local address 127.0.0.1:56728)
0927 22:09:15.863311 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:15.863398 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260927 22:09:15.863627 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:15.860903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59532 (local address 127.0.97.3:45835)
0927 22:09:15.861109 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:15.861115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:15.861138 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:09:15.861296 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:15.861301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:15.861370 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:15.861502 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:15.862893 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:15.862901 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:15.862910 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:09:15.862932 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:15.862969 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:09:15.862985 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:15.863134 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:15.863257 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 22:09:15.863316 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 22:09:15.864394 5486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:15.944244 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:15.943731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44563 (local address 127.0.0.1:56736)
0927 22:09:15.943988 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:15.944083 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 22:09:15.945523 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:15.945092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44563 (local address 127.0.0.1:56740)
0927 22:09:15.945300 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:15.945376 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260927 22:09:15.982138 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 5364
I20260927 22:09:16.000651 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 5014
I20260927 22:09:16.001812 5075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:16.105374 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 5014
2026-09-27T22:09:16Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2985 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 22:09:16.145805 388 mini_oidc.cc:152] Starting JWKS server
I20260927 22:09:16.148461 388 webserver.cc:460] Webserver started at https://127.0.0.1:36489/ using document root <none> and password file <none>
I20260927 22:09:16.148769 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 22:09:16.149415 388 webserver.cc:460] Webserver started at http://127.0.0.1:44575/ using document root <none> and password file <none>
2026-09-27T22:09:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:09:16Z Disabled control of system clock
I20260927 22:09:16.175853 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33453
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33453
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36489/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546912983837-388-0/testchainca.cert with env {}
W20260927 22:09:16.509074 5547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:16.509478 5547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:16.509660 5547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:16.517203 5547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:09:16.517374 5547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:16.517467 5547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:16.517592 5547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:09:16.517683 5547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:09:16.517781 5547 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:09:16.517879 5547 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36489/jwks.json
I20260927 22:09:16.527992 5547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33453
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36489/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546912983837-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:16.529415 5547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:16.530804 5547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:16.543181 5553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:16.543201 5552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:16.543288 5555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:16.543963 5547 server_base.cc:1034] running on GCE node
I20260927 22:09:16.544723 5547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:16.546108 5547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:16.547328 5547 hybrid_clock.cc:648] HybridClock initialized: now 1790546956547282 us; error 63 us; skew 500 ppm
I20260927 22:09:16.547686 5547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:09:16.549959 5547 webserver.cc:460] Webserver started at http://127.0.97.62:45207/ using document root <none> and password file <none>
I20260927 22:09:16.550699 5547 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:16.550827 5547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:16.551126 5547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:16.553982 5547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "443f31d018524a4194e8c2c003aadfc8"
format_stamp: "Formatted at 2026-09-27 22:09:16 on dist-test-slave-mp03"
server_key: "a7432e6a8d34b9e19a025842eb2d8559"
server_key_iv: "9bce763fd496f64a4500e4a6fa082826"
server_key_version: "encryptionkey@0"
I20260927 22:09:16.554862 5547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "443f31d018524a4194e8c2c003aadfc8"
format_stamp: "Formatted at 2026-09-27 22:09:16 on dist-test-slave-mp03"
server_key: "a7432e6a8d34b9e19a025842eb2d8559"
server_key_iv: "9bce763fd496f64a4500e4a6fa082826"
server_key_version: "encryptionkey@0"
I20260927 22:09:16.560027 5547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 22:09:16.563511 5561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:16.564844 5547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:09:16.565032 5547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "443f31d018524a4194e8c2c003aadfc8"
format_stamp: "Formatted at 2026-09-27 22:09:16 on dist-test-slave-mp03"
server_key: "a7432e6a8d34b9e19a025842eb2d8559"
server_key_iv: "9bce763fd496f64a4500e4a6fa082826"
server_key_version: "encryptionkey@0"
I20260927 22:09:16.565244 5547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:16.590323 5547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:16.591259 5547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:16.591550 5547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:16.610299 5547 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33453
I20260927 22:09:16.610317 5612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33453 every 8 connection(s)
I20260927 22:09:16.611922 5547 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:09:16.613802 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 5547
I20260927 22:09:16.614046 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:09:16.614786 388 external_mini_cluster.cc:1428] Setting key 8d690440a71e93cbb0287268c107af73
I20260927 22:09:16.616511 5613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:16.628404 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:16.616756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57334 (local address 127.0.97.62:33453)
0927 22:09:16.617683 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:16.617694 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:16.617734 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:09:16.619337 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:16.619359 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:16.619821 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:16.620164 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:16.620823 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:16.622225 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:16.624119 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:16.624740 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:16.624866 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:16.626121 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:16.626127 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:16.626140 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:09:16.626744 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:16.626819 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:09:16.626841 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:16.626931 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:16.627147 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 22:09:16.627487 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":327,"threads_started":1}
I20260927 22:09:16.629390 5613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443f31d018524a4194e8c2c003aadfc8: Bootstrap starting.
I20260927 22:09:16.632192 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:09:16.632733 5613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 443f31d018524a4194e8c2c003aadfc8: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:16.633687 5613 log.cc:824] T 00000000000000000000000000000000 P 443f31d018524a4194e8c2c003aadfc8: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:16.636348 5613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443f31d018524a4194e8c2c003aadfc8: No bootstrap required, opened a new log
I20260927 22:09:16.643196 5613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 443f31d018524a4194e8c2c003aadfc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443f31d018524a4194e8c2c003aadfc8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33453 } }
I20260927 22:09:16.643551 5613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 443f31d018524a4194e8c2c003aadfc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:16.643656 5613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 443f31d018524a4194e8c2c003aadfc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443f31d018524a4194e8c2c003aadfc8, State: Initialized, Role: FOLLOWER
I20260927 22:09:16.644264 5613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 443f31d018524a4194e8c2c003aadfc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443f31d018524a4194e8c2c003aadfc8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33453 } }
I20260927 22:09:16.644492 5613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 443f31d018524a4194e8c2c003aadfc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:09:16.644624 5613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 443f31d018524a4194e8c2c003aadfc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:09:16.644788 5613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 443f31d018524a4194e8c2c003aadfc8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:16.647105 5613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 443f31d018524a4194e8c2c003aadfc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443f31d018524a4194e8c2c003aadfc8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33453 } }
I20260927 22:09:16.647601 5613 leader_election.cc:302] T 00000000000000000000000000000000 P 443f31d018524a4194e8c2c003aadfc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 443f31d018524a4194e8c2c003aadfc8; no voters:
I20260927 22:09:16.648138 5613 leader_election.cc:288] T 00000000000000000000000000000000 P 443f31d018524a4194e8c2c003aadfc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:09:16.648288 5618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 443f31d018524a4194e8c2c003aadfc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:09:16.648934 5618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 443f31d018524a4194e8c2c003aadfc8 [term 1 LEADER]: Becoming Leader. State: Replica: 443f31d018524a4194e8c2c003aadfc8, State: Running, Role: LEADER
I20260927 22:09:16.649477 5618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 443f31d018524a4194e8c2c003aadfc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443f31d018524a4194e8c2c003aadfc8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33453 } }
I20260927 22:09:16.650180 5613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 443f31d018524a4194e8c2c003aadfc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:09:16.652364 5620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443f31d018524a4194e8c2c003aadfc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 443f31d018524a4194e8c2c003aadfc8. Latest consensus state: current_term: 1 leader_uuid: "443f31d018524a4194e8c2c003aadfc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443f31d018524a4194e8c2c003aadfc8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33453 } } }
I20260927 22:09:16.652798 5620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443f31d018524a4194e8c2c003aadfc8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:16.653537 5619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443f31d018524a4194e8c2c003aadfc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "443f31d018524a4194e8c2c003aadfc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443f31d018524a4194e8c2c003aadfc8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33453 } } }
I20260927 22:09:16.654013 5619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443f31d018524a4194e8c2c003aadfc8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:16.654551 5627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:09:16.659528 5627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:09:16.668130 5627 catalog_manager.cc:1347] Generated new cluster ID: 4bb10ff2c84644b4868ee3949424033c
I20260927 22:09:16.668255 5627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:09:16.690511 5627 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:09:16.691848 5627 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:09:16.699219 5627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 443f31d018524a4194e8c2c003aadfc8: Generated new TSK 0
I20260927 22:09:16.700263 5627 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 22:09:16.745210 5643 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 22:09:16.745393 5643 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 22:09:16.746455 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:16.725062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57340 (local address 127.0.97.62:33453)
0927 22:09:16.725277 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:16.725289 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:16.725735 (+ 446us) server_negotiation.cc:407] Connection header received
0927 22:09:16.726023 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:16.726029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:16.726133 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:16.726291 (+ 158us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:09:16.726824 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:16.727956 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:16.729345 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:16.730055 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:16.730171 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:16.730705 (+ 534us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:09:16.745768 (+ 15063us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 22:09:16.746136 (+ 368us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57340: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":85}
W20260927 22:09:16.746587 5616 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57340: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 22:09:16.748124 5642 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:16.724949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33453 (local address 127.0.0.1:57340)
0927 22:09:16.725632 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:16.725649 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:16.725796 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:16.726320 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:16.726323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:16.726332 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:09:16.726688 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:16.726696 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:16.728098 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:16.728102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:16.729208 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:16.729218 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:16.730174 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:16.730212 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:16.730334 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:16.730391 (+ 57us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:09:16.746073 (+ 15682us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 22:09:16.746338 (+ 265us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:33453: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":363,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:33453: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 22:09:16.749637 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 5547
I20260927 22:09:16.750885 5608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:16.843413 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 5547
2026-09-27T22:09:16Z chronyd exiting
I20260927 22:09:16.893751 388 test_util.cc:175] -----------------------------------------------
I20260927 22:09:16.893899 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546912983837-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (752 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 22:09:16.897526 388 mini_oidc.cc:152] Starting JWKS server
I20260927 22:09:16.899351 388 webserver.cc:460] Webserver started at https://127.0.0.1:35853/ using document root <none> and password file <none>
I20260927 22:09:16.899528 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 22:09:16.900110 388 webserver.cc:460] Webserver started at http://127.0.0.1:42807/ using document root <none> and password file <none>
2026-09-27T22:09:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:09:16Z Disabled control of system clock
I20260927 22:09:16.926400 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37303
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37303
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35853/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546912983837-388-0/testchainca.cert with env {}
W20260927 22:09:17.259788 5650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:17.260130 5650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:17.260277 5650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:17.267577 5650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:09:17.267722 5650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:17.267809 5650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:17.267918 5650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:09:17.267992 5650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:09:17.268079 5650 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:09:17.268155 5650 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35853/jwks.json
I20260927 22:09:17.278213 5650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37303
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35853/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546912983837-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:17.279572 5650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:17.280871 5650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:17.292342 5655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:17.292404 5656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:17.292456 5658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:17.293129 5650 server_base.cc:1034] running on GCE node
I20260927 22:09:17.293991 5650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:17.295485 5650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:17.296707 5650 hybrid_clock.cc:648] HybridClock initialized: now 1790546957296650 us; error 56 us; skew 500 ppm
I20260927 22:09:17.297057 5650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:09:17.299394 5650 webserver.cc:460] Webserver started at http://127.0.97.62:40861/ using document root <none> and password file <none>
I20260927 22:09:17.300100 5650 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:17.300230 5650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:17.300558 5650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:17.303422 5650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "ccde810f352347e287da45c92ed5b0a0"
format_stamp: "Formatted at 2026-09-27 22:09:17 on dist-test-slave-mp03"
server_key: "b3334acd6515a5df3f5b46a64511b94d"
server_key_iv: "55112e53514fd06096b4fc1a62ed622f"
server_key_version: "encryptionkey@0"
I20260927 22:09:17.304215 5650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "ccde810f352347e287da45c92ed5b0a0"
format_stamp: "Formatted at 2026-09-27 22:09:17 on dist-test-slave-mp03"
server_key: "b3334acd6515a5df3f5b46a64511b94d"
server_key_iv: "55112e53514fd06096b4fc1a62ed622f"
server_key_version: "encryptionkey@0"
I20260927 22:09:17.309157 5650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:09:17.312631 5664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:17.314055 5650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:09:17.314194 5650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "ccde810f352347e287da45c92ed5b0a0"
format_stamp: "Formatted at 2026-09-27 22:09:17 on dist-test-slave-mp03"
server_key: "b3334acd6515a5df3f5b46a64511b94d"
server_key_iv: "55112e53514fd06096b4fc1a62ed622f"
server_key_version: "encryptionkey@0"
I20260927 22:09:17.314369 5650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:17.328611 5650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:17.329362 5650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:17.329589 5650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:17.347510 5650 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37303
I20260927 22:09:17.347534 5715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37303 every 8 connection(s)
I20260927 22:09:17.348834 5650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:09:17.352598 5716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:17.355021 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 5650
I20260927 22:09:17.355209 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:09:17.355895 388 external_mini_cluster.cc:1428] Setting key 991960e74f3f8ff515716c8c6f3b9367
I20260927 22:09:17.365713 5716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccde810f352347e287da45c92ed5b0a0: Bootstrap starting.
I20260927 22:09:17.369633 5716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccde810f352347e287da45c92ed5b0a0: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:17.370990 5716 log.cc:824] T 00000000000000000000000000000000 P ccde810f352347e287da45c92ed5b0a0: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:17.371433 5719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:17.357800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54042 (local address 127.0.97.62:37303)
0927 22:09:17.358880 (+ 1080us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:17.358894 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:17.358942 (+ 48us) server_negotiation.cc:407] Connection header received
0927 22:09:17.361269 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:17.361304 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:17.361933 (+ 629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:17.362418 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:17.363014 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:17.364928 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:17.366283 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:17.367328 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:17.367529 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:17.368559 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:17.368569 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:17.368587 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:09:17.369306 (+ 719us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:17.369421 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:09:17.369449 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:17.369566 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:17.369801 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 22:09:17.370172 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":333,"threads_started":1}
I20260927 22:09:17.374068 5716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccde810f352347e287da45c92ed5b0a0: No bootstrap required, opened a new log
I20260927 22:09:17.375176 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:09:17.381506 5716 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccde810f352347e287da45c92ed5b0a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccde810f352347e287da45c92ed5b0a0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37303 } }
I20260927 22:09:17.381873 5716 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccde810f352347e287da45c92ed5b0a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:17.381978 5716 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccde810f352347e287da45c92ed5b0a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccde810f352347e287da45c92ed5b0a0, State: Initialized, Role: FOLLOWER
I20260927 22:09:17.382583 5716 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccde810f352347e287da45c92ed5b0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccde810f352347e287da45c92ed5b0a0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37303 } }
I20260927 22:09:17.382874 5716 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccde810f352347e287da45c92ed5b0a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:09:17.383023 5716 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccde810f352347e287da45c92ed5b0a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:09:17.383208 5716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccde810f352347e287da45c92ed5b0a0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:17.385564 5716 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccde810f352347e287da45c92ed5b0a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccde810f352347e287da45c92ed5b0a0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37303 } }
I20260927 22:09:17.386066 5716 leader_election.cc:302] T 00000000000000000000000000000000 P ccde810f352347e287da45c92ed5b0a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccde810f352347e287da45c92ed5b0a0; no voters:
I20260927 22:09:17.386698 5716 leader_election.cc:288] T 00000000000000000000000000000000 P ccde810f352347e287da45c92ed5b0a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:09:17.386861 5721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccde810f352347e287da45c92ed5b0a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:09:17.387573 5721 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccde810f352347e287da45c92ed5b0a0 [term 1 LEADER]: Becoming Leader. State: Replica: ccde810f352347e287da45c92ed5b0a0, State: Running, Role: LEADER
I20260927 22:09:17.388124 5721 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccde810f352347e287da45c92ed5b0a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccde810f352347e287da45c92ed5b0a0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37303 } }
I20260927 22:09:17.388863 5716 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccde810f352347e287da45c92ed5b0a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:09:17.390770 5722 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccde810f352347e287da45c92ed5b0a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccde810f352347e287da45c92ed5b0a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccde810f352347e287da45c92ed5b0a0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37303 } } }
I20260927 22:09:17.391152 5722 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccde810f352347e287da45c92ed5b0a0 [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:17.391288 5723 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccde810f352347e287da45c92ed5b0a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccde810f352347e287da45c92ed5b0a0. Latest consensus state: current_term: 1 leader_uuid: "ccde810f352347e287da45c92ed5b0a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccde810f352347e287da45c92ed5b0a0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37303 } } }
I20260927 22:09:17.391582 5723 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccde810f352347e287da45c92ed5b0a0 [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:17.393728 5728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:09:17.399358 5728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:09:17.407753 5728 catalog_manager.cc:1347] Generated new cluster ID: 736cd157e64a46b682bd3f966e4140da
I20260927 22:09:17.407886 5728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:09:17.422333 5728 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:09:17.424129 5728 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:09:17.437378 5728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccde810f352347e287da45c92ed5b0a0: Generated new TSK 0
I20260927 22:09:17.438206 5728 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 22:09:17.473397 5746 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 22:09:17.473534 5746 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 22:09:17.474802 5719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:17.454508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54052 (local address 127.0.97.62:37303)
0927 22:09:17.454730 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:17.454737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:17.455261 (+ 524us) server_negotiation.cc:407] Connection header received
0927 22:09:17.455469 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:17.455475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:17.455573 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:17.455734 (+ 161us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:09:17.456289 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:17.457427 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:17.458838 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:17.459513 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:17.459635 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:17.460124 (+ 489us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:09:17.474076 (+ 13952us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35853/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 22:09:17.474485 (+ 409us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54052: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35853/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 22:09:17.474901 5745 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:17.454458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37303 (local address 127.0.0.1:54052)
0927 22:09:17.455149 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:17.455169 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:17.455289 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:17.455756 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:17.455759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:17.455768 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:09:17.456079 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:17.456093 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:17.457561 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:17.457565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:17.458705 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:17.458716 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:17.459632 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:17.459687 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:17.459817 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:17.459848 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:09:17.474350 (+ 14502us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35853/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 22:09:17.474539 (+ 189us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:37303: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35853/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":397,"threads_started":1}
W20260927 22:09:17.474920 5719 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54052: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35853/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260927 22:09:17.476325 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 5650
I20260927 22:09:17.477470 5711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:17.567042 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 5650
2026-09-27T22:09:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (725 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 22:09:17.623461 388 mini_oidc.cc:152] Starting JWKS server
I20260927 22:09:17.625512 388 webserver.cc:460] Webserver started at https://127.0.0.1:39365/ using document root <none> and password file <none>
I20260927 22:09:17.625860 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 22:09:17.626533 388 webserver.cc:460] Webserver started at http://127.0.0.1:41053/ using document root <none> and password file <none>
2026-09-27T22:09:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:09:17Z Disabled control of system clock
I20260927 22:09:17.652041 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43373
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43373
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39365/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 22:09:17.967645 5753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:17.967937 5753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:17.968122 5753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:17.975361 5753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:09:17.975499 5753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:17.975587 5753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:17.975690 5753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:09:17.975766 5753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:09:17.975852 5753 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:09:17.975927 5753 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39365/jwks.json
I20260927 22:09:17.985116 5753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43373
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39365/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:17.986402 5753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:17.988215 5753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:17.999701 5758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:17.999771 5759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:18.000432 5761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:18.000524 5753 server_base.cc:1034] running on GCE node
I20260927 22:09:18.001317 5753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:18.002722 5753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:18.003960 5753 hybrid_clock.cc:648] HybridClock initialized: now 1790546958003911 us; error 42 us; skew 500 ppm
I20260927 22:09:18.004341 5753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:09:18.006533 5753 webserver.cc:460] Webserver started at http://127.0.97.62:33755/ using document root <none> and password file <none>
I20260927 22:09:18.007230 5753 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:18.007354 5753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:18.007676 5753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:18.010504 5753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "2ba300c36a6d4558883e8a3a97d7f8ca"
format_stamp: "Formatted at 2026-09-27 22:09:18 on dist-test-slave-mp03"
server_key: "1e2ae0f6b3970b9c21593f043c8aa9d8"
server_key_iv: "8ca3ea43ea9558f5a7a388ed048cbbdc"
server_key_version: "encryptionkey@0"
I20260927 22:09:18.011328 5753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "2ba300c36a6d4558883e8a3a97d7f8ca"
format_stamp: "Formatted at 2026-09-27 22:09:18 on dist-test-slave-mp03"
server_key: "1e2ae0f6b3970b9c21593f043c8aa9d8"
server_key_iv: "8ca3ea43ea9558f5a7a388ed048cbbdc"
server_key_version: "encryptionkey@0"
I20260927 22:09:18.016215 5753 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:09:18.019608 5767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:18.020857 5753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:09:18.021031 5753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "2ba300c36a6d4558883e8a3a97d7f8ca"
format_stamp: "Formatted at 2026-09-27 22:09:18 on dist-test-slave-mp03"
server_key: "1e2ae0f6b3970b9c21593f043c8aa9d8"
server_key_iv: "8ca3ea43ea9558f5a7a388ed048cbbdc"
server_key_version: "encryptionkey@0"
I20260927 22:09:18.021255 5753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:18.039683 5753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:18.040511 5753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:18.040813 5753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:18.058840 5753 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43373
I20260927 22:09:18.058878 5818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43373 every 8 connection(s)
I20260927 22:09:18.060187 5753 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:09:18.063910 5819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:18.069288 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 5753
I20260927 22:09:18.069501 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:09:18.070120 388 external_mini_cluster.cc:1428] Setting key 3400cadc99bd21b60b73152e16a083f2
I20260927 22:09:18.074784 5819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba300c36a6d4558883e8a3a97d7f8ca: Bootstrap starting.
I20260927 22:09:18.078070 5819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ba300c36a6d4558883e8a3a97d7f8ca: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:18.079320 5819 log.cc:824] T 00000000000000000000000000000000 P 2ba300c36a6d4558883e8a3a97d7f8ca: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:18.082399 5819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba300c36a6d4558883e8a3a97d7f8ca: No bootstrap required, opened a new log
I20260927 22:09:18.084431 5822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:18.072021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38904 (local address 127.0.97.62:43373)
0927 22:09:18.073068 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:18.073078 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:18.073119 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:09:18.075327 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:18.075349 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:18.075797 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:18.076115 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:18.076728 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:18.077922 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:18.079730 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:18.080401 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:18.080535 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:18.081664 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:18.081674 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:18.081691 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:09:18.082345 (+ 654us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:18.082454 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:09:18.082479 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:18.082666 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:18.082881 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 22:09:18.083220 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":409,"threads_started":1}
I20260927 22:09:18.088294 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:09:18.091255 5819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ba300c36a6d4558883e8a3a97d7f8ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba300c36a6d4558883e8a3a97d7f8ca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43373 } }
I20260927 22:09:18.091706 5819 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ba300c36a6d4558883e8a3a97d7f8ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:18.091859 5819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ba300c36a6d4558883e8a3a97d7f8ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba300c36a6d4558883e8a3a97d7f8ca, State: Initialized, Role: FOLLOWER
I20260927 22:09:18.092419 5819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ba300c36a6d4558883e8a3a97d7f8ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba300c36a6d4558883e8a3a97d7f8ca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43373 } }
I20260927 22:09:18.092639 5819 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ba300c36a6d4558883e8a3a97d7f8ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:09:18.092788 5819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ba300c36a6d4558883e8a3a97d7f8ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:09:18.092948 5819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ba300c36a6d4558883e8a3a97d7f8ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:18.095459 5819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ba300c36a6d4558883e8a3a97d7f8ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba300c36a6d4558883e8a3a97d7f8ca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43373 } }
I20260927 22:09:18.095957 5819 leader_election.cc:302] T 00000000000000000000000000000000 P 2ba300c36a6d4558883e8a3a97d7f8ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ba300c36a6d4558883e8a3a97d7f8ca; no voters:
I20260927 22:09:18.096542 5819 leader_election.cc:288] T 00000000000000000000000000000000 P 2ba300c36a6d4558883e8a3a97d7f8ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:09:18.096701 5824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ba300c36a6d4558883e8a3a97d7f8ca [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:09:18.097365 5824 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ba300c36a6d4558883e8a3a97d7f8ca [term 1 LEADER]: Becoming Leader. State: Replica: 2ba300c36a6d4558883e8a3a97d7f8ca, State: Running, Role: LEADER
I20260927 22:09:18.097887 5824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ba300c36a6d4558883e8a3a97d7f8ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba300c36a6d4558883e8a3a97d7f8ca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43373 } }
I20260927 22:09:18.098582 5819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ba300c36a6d4558883e8a3a97d7f8ca [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:09:18.100698 5826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ba300c36a6d4558883e8a3a97d7f8ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ba300c36a6d4558883e8a3a97d7f8ca. Latest consensus state: current_term: 1 leader_uuid: "2ba300c36a6d4558883e8a3a97d7f8ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba300c36a6d4558883e8a3a97d7f8ca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43373 } } }
I20260927 22:09:18.101111 5826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ba300c36a6d4558883e8a3a97d7f8ca [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:18.101178 5825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ba300c36a6d4558883e8a3a97d7f8ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ba300c36a6d4558883e8a3a97d7f8ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba300c36a6d4558883e8a3a97d7f8ca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43373 } } }
I20260927 22:09:18.101502 5825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ba300c36a6d4558883e8a3a97d7f8ca [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:18.103641 5831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:09:18.108493 5831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:09:18.120949 5831 catalog_manager.cc:1347] Generated new cluster ID: 651e8561391944ddb10eb19b0581949f
I20260927 22:09:18.121071 5831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:09:18.134919 5831 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:09:18.136411 5831 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:09:18.152331 5831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ba300c36a6d4558883e8a3a97d7f8ca: Generated new TSK 0
I20260927 22:09:18.153180 5831 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:09:18.173175 5822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:18.164868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38906 (local address 127.0.97.62:43373)
0927 22:09:18.165066 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:18.165072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:18.165601 (+ 529us) server_negotiation.cc:407] Connection header received
0927 22:09:18.165824 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:18.165830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:18.165921 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:18.166078 (+ 157us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:09:18.166487 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:18.167640 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:18.168960 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:18.169653 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:18.169792 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:18.170209 (+ 417us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:09:18.172552 (+ 2343us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39365/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 22:09:18.172873 (+ 321us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38906: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39365/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 22:09:18.173297 5822 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38906: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39365/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 22:09:18.175036 5848 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:18.164818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43373 (local address 127.0.0.1:38906)
0927 22:09:18.165504 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:18.165520 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:18.165638 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:18.166096 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:18.166100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:18.166109 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:09:18.166345 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:18.166352 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:18.167767 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:18.167771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:18.168826 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:18.168837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:18.169764 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:18.169821 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:18.169939 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:18.169968 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:09:18.172770 (+ 2802us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39365/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 22:09:18.174638 (+ 1868us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:43373: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39365/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":1554,"thread_start_us":398,"threads_started":1}
I20260927 22:09:18.175952 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 5753
I20260927 22:09:18.177059 5814 generic_service.cc:196] Checking for leaks (request via RPC)
E20260927 22:09:18.185199 5849 webserver.cc:533] Webserver: sslize failed:
E20260927 22:09:18.185281 5849 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 22:09:18.267400 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 5753
2026-09-27T22:09:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (698 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T22:09:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:09:18Z Disabled control of system clock
I20260927 22:09:18.347231 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34607
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34607
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546912983837-388-0/testchainca.cert with env {}
W20260927 22:09:18.662307 5854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:18.662647 5854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:18.662755 5854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:18.669970 5854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:09:18.670141 5854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:18.670230 5854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:18.670342 5854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:09:18.670426 5854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:09:18.670512 5854 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:09:18.670570 5854 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 22:09:18.680505 5854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34607
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546912983837-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:18.681878 5854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:18.683269 5854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:18.695041 5859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:18.695089 5862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:18.695089 5860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:18.695979 5854 server_base.cc:1034] running on GCE node
I20260927 22:09:18.696740 5854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:18.698153 5854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:18.699427 5854 hybrid_clock.cc:648] HybridClock initialized: now 1790546958699381 us; error 57 us; skew 500 ppm
I20260927 22:09:18.699841 5854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:09:18.702262 5854 webserver.cc:460] Webserver started at http://127.0.97.62:36983/ using document root <none> and password file <none>
I20260927 22:09:18.703055 5854 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:18.703198 5854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:18.703521 5854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:18.706444 5854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "5ac3c5c4dc7144f688ac0b687bf2d048"
format_stamp: "Formatted at 2026-09-27 22:09:18 on dist-test-slave-mp03"
server_key: "12c8c3f37ebb2dab90461dacad92ab80"
server_key_iv: "145c7ea3fb95d8f08c2e24ca6329ec11"
server_key_version: "encryptionkey@0"
I20260927 22:09:18.707325 5854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "5ac3c5c4dc7144f688ac0b687bf2d048"
format_stamp: "Formatted at 2026-09-27 22:09:18 on dist-test-slave-mp03"
server_key: "12c8c3f37ebb2dab90461dacad92ab80"
server_key_iv: "145c7ea3fb95d8f08c2e24ca6329ec11"
server_key_version: "encryptionkey@0"
I20260927 22:09:18.712402 5854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:09:18.715803 5868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:18.717070 5854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:09:18.717255 5854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "5ac3c5c4dc7144f688ac0b687bf2d048"
format_stamp: "Formatted at 2026-09-27 22:09:18 on dist-test-slave-mp03"
server_key: "12c8c3f37ebb2dab90461dacad92ab80"
server_key_iv: "145c7ea3fb95d8f08c2e24ca6329ec11"
server_key_version: "encryptionkey@0"
I20260927 22:09:18.717466 5854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:18.735798 5854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:18.736706 5854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:18.737027 5854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:18.756636 5854 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34607
I20260927 22:09:18.756659 5919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34607 every 8 connection(s)
I20260927 22:09:18.758118 5854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:09:18.761886 5920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:18.765129 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 5854
I20260927 22:09:18.765333 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:09:18.766119 388 external_mini_cluster.cc:1428] Setting key 38e2e9d954910781ba6c378687b881aa
I20260927 22:09:18.773986 5920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ac3c5c4dc7144f688ac0b687bf2d048: Bootstrap starting.
I20260927 22:09:18.776976 5920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ac3c5c4dc7144f688ac0b687bf2d048: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:18.778014 5920 log.cc:824] T 00000000000000000000000000000000 P 5ac3c5c4dc7144f688ac0b687bf2d048: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:18.779563 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:18.768108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34478 (local address 127.0.97.62:34607)
0927 22:09:18.769061 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:18.769072 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:18.769111 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:09:18.770761 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:18.770783 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:18.771283 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:18.771683 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:18.772276 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:18.773482 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:18.774829 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:18.775508 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:18.775705 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:18.776845 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:18.776852 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:18.776864 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:09:18.777480 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:18.777566 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:09:18.777585 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:18.777706 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:18.777927 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 22:09:18.778326 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":322,"threads_started":1}
I20260927 22:09:18.780948 5920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ac3c5c4dc7144f688ac0b687bf2d048: No bootstrap required, opened a new log
I20260927 22:09:18.783226 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:09:18.788614 5920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ac3c5c4dc7144f688ac0b687bf2d048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ac3c5c4dc7144f688ac0b687bf2d048" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34607 } }
I20260927 22:09:18.789119 5920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ac3c5c4dc7144f688ac0b687bf2d048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:18.789252 5920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ac3c5c4dc7144f688ac0b687bf2d048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ac3c5c4dc7144f688ac0b687bf2d048, State: Initialized, Role: FOLLOWER
I20260927 22:09:18.789850 5920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ac3c5c4dc7144f688ac0b687bf2d048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ac3c5c4dc7144f688ac0b687bf2d048" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34607 } }
I20260927 22:09:18.790079 5920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ac3c5c4dc7144f688ac0b687bf2d048 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:09:18.790216 5920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ac3c5c4dc7144f688ac0b687bf2d048 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:09:18.790391 5920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ac3c5c4dc7144f688ac0b687bf2d048 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:18.792505 5920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ac3c5c4dc7144f688ac0b687bf2d048 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ac3c5c4dc7144f688ac0b687bf2d048" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34607 } }
I20260927 22:09:18.793042 5920 leader_election.cc:302] T 00000000000000000000000000000000 P 5ac3c5c4dc7144f688ac0b687bf2d048 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ac3c5c4dc7144f688ac0b687bf2d048; no voters:
I20260927 22:09:18.793712 5920 leader_election.cc:288] T 00000000000000000000000000000000 P 5ac3c5c4dc7144f688ac0b687bf2d048 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:09:18.793890 5925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ac3c5c4dc7144f688ac0b687bf2d048 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:09:18.794556 5925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ac3c5c4dc7144f688ac0b687bf2d048 [term 1 LEADER]: Becoming Leader. State: Replica: 5ac3c5c4dc7144f688ac0b687bf2d048, State: Running, Role: LEADER
I20260927 22:09:18.795083 5925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ac3c5c4dc7144f688ac0b687bf2d048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ac3c5c4dc7144f688ac0b687bf2d048" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34607 } }
I20260927 22:09:18.795785 5920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ac3c5c4dc7144f688ac0b687bf2d048 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:09:18.797644 5926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ac3c5c4dc7144f688ac0b687bf2d048 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ac3c5c4dc7144f688ac0b687bf2d048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ac3c5c4dc7144f688ac0b687bf2d048" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34607 } } }
I20260927 22:09:18.798025 5926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ac3c5c4dc7144f688ac0b687bf2d048 [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:18.798072 5927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ac3c5c4dc7144f688ac0b687bf2d048 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ac3c5c4dc7144f688ac0b687bf2d048. Latest consensus state: current_term: 1 leader_uuid: "5ac3c5c4dc7144f688ac0b687bf2d048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ac3c5c4dc7144f688ac0b687bf2d048" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34607 } } }
I20260927 22:09:18.798403 5927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ac3c5c4dc7144f688ac0b687bf2d048 [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:18.799109 5930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:09:18.806440 5930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:09:18.815073 5930 catalog_manager.cc:1347] Generated new cluster ID: 7e5671283c1e4abb90fb29b7ed1b6756
I20260927 22:09:18.815228 5930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:09:18.825023 5930 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:09:18.826875 5930 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:09:18.836308 5930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ac3c5c4dc7144f688ac0b687bf2d048: Generated new TSK 0
I20260927 22:09:18.837409 5930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:09:19.373914 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:18.860508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34490 (local address 127.0.97.62:34607)
0927 22:09:18.860757 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:18.860763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:18.861115 (+ 352us) server_negotiation.cc:407] Connection header received
0927 22:09:18.861332 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:18.861336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:18.861406 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:18.861534 (+ 128us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:09:18.862073 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:18.862839 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:18.864850 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:18.865272 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:18.865357 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:18.865931 (+ 574us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:09:19.373366 (+507435us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 22:09:19.373700 (+ 334us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34490: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":77}
W20260927 22:09:19.374058 5923 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34490: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 22:09:19.374207 5949 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:18.860417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34607 (local address 127.0.0.1:34490)
0927 22:09:18.861003 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:18.861024 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:18.861181 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:18.861587 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:18.861592 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:18.861605 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:09:18.861926 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:18.861938 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:18.862987 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:18.862992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:18.864693 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:18.864708 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:18.865386 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:18.865442 (+ 56us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:18.865643 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:18.865690 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:09:19.373639 (+507949us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 22:09:19.373862 (+ 223us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:34607: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":298,"threads_started":1}
I20260927 22:09:19.375597 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 5854
I20260927 22:09:19.376629 5915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:19.475347 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 5854
2026-09-27T22:09:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1189 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (407 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (405 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:09:20 dist-test-slave-mp03 krb5kdc[5964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:09:20 dist-test-slave-mp03 krb5kdc[5964](info): set up 2 sockets
Sep 27 22:09:20 dist-test-slave-mp03 krb5kdc[5964](info): commencing operation
krb5kdc: starting...
W20260927 22:09:21.411304 388 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:09:21 dist-test-slave-mp03 krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:09:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:09:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:09:21.616201 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36765
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:21.935303 5977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:21.935596 5977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:21.935695 5977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:21.942795 5977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:09:21.942910 5977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:21.942955 5977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:21.943003 5977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:09:21.943039 5977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:09:21.953101 5977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:21.954487 5977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:21.955899 5977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:21.967742 5982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:21.967808 5985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:21.967828 5983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:21.968369 5977 server_base.cc:1034] running on GCE node
I20260927 22:09:21.969190 5977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:21.970535 5977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:21.971787 5977 hybrid_clock.cc:648] HybridClock initialized: now 1790546961971731 us; error 51 us; skew 500 ppm
I20260927 22:09:21.972133 5977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:21 dist-test-slave-mp03 krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:21.975311 5977 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:09:21.976696 5977 webserver.cc:460] Webserver started at http://127.0.97.62:46423/ using document root <none> and password file <none>
I20260927 22:09:21.977311 5977 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:21.977406 5977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:21.977653 5977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:21.980540 5977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "7c728d90c4154315866d4a69c26d5b15"
format_stamp: "Formatted at 2026-09-27 22:09:21 on dist-test-slave-mp03"
server_key: "767f6b02f198376402e896029c5f4693"
server_key_iv: "7e6ebe35fc6faa22a54a2472ffa1da02"
server_key_version: "encryptionkey@0"
I20260927 22:09:21.981226 5977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "7c728d90c4154315866d4a69c26d5b15"
format_stamp: "Formatted at 2026-09-27 22:09:21 on dist-test-slave-mp03"
server_key: "767f6b02f198376402e896029c5f4693"
server_key_iv: "7e6ebe35fc6faa22a54a2472ffa1da02"
server_key_version: "encryptionkey@0"
I20260927 22:09:21.985987 5977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 22:09:21.989372 5992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:21.990738 5977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 22:09:21.990865 5977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "7c728d90c4154315866d4a69c26d5b15"
format_stamp: "Formatted at 2026-09-27 22:09:21 on dist-test-slave-mp03"
server_key: "767f6b02f198376402e896029c5f4693"
server_key_iv: "7e6ebe35fc6faa22a54a2472ffa1da02"
server_key_version: "encryptionkey@0"
I20260927 22:09:21.991015 5977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:22.004112 5977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:22.008399 5977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:22.008626 5977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:22.026515 5977 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36765
I20260927 22:09:22.026535 6043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36765 every 8 connection(s)
I20260927 22:09:22.027956 5977 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:09:22.031517 6044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:22.035211 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 5977
I20260927 22:09:22.035396 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:09:22.036043 388 external_mini_cluster.cc:1428] Setting key 5c554128dbb21d4e28c2bc28b6756cb9
I20260927 22:09:22.043501 6044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c728d90c4154315866d4a69c26d5b15: Bootstrap starting.
I20260927 22:09:22.047399 6044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c728d90c4154315866d4a69c26d5b15: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:22.048601 6044 log.cc:824] T 00000000000000000000000000000000 P 7c728d90c4154315866d4a69c26d5b15: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:22.052147 6044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c728d90c4154315866d4a69c26d5b15: No bootstrap required, opened a new log
Sep 27 22:09:22 dist-test-slave-mp03 krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:09:22.059861 6044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c728d90c4154315866d4a69c26d5b15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c728d90c4154315866d4a69c26d5b15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36765 } }
I20260927 22:09:22.060261 6044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c728d90c4154315866d4a69c26d5b15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:22.060381 6044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c728d90c4154315866d4a69c26d5b15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c728d90c4154315866d4a69c26d5b15, State: Initialized, Role: FOLLOWER
I20260927 22:09:22.060854 6047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:22.038137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48566 (local address 127.0.97.62:36765)
0927 22:09:22.038934 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:22.038950 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:22.039003 (+ 53us) server_negotiation.cc:407] Connection header received
0927 22:09:22.041735 (+ 2732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:22.041762 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:22.042238 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:22.042660 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:22.044493 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:22.045891 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:22.047767 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:22.048491 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:22.048629 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:22.053405 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:22.053413 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:22.053430 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:22.053469 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:22.056682 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:22.057371 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:22.057377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:22.057384 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:22.057547 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:22.057970 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:22.057974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:22.057976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:22.058522 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:22.058862 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:22.059284 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 22:09:22.059695 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":318,"threads_started":1}
I20260927 22:09:22.060990 6044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c728d90c4154315866d4a69c26d5b15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c728d90c4154315866d4a69c26d5b15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36765 } }
I20260927 22:09:22.061314 6044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c728d90c4154315866d4a69c26d5b15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:09:22.061458 6044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c728d90c4154315866d4a69c26d5b15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:09:22.061636 6044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c728d90c4154315866d4a69c26d5b15 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:22.064270 6044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c728d90c4154315866d4a69c26d5b15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c728d90c4154315866d4a69c26d5b15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36765 } }
I20260927 22:09:22.064671 6044 leader_election.cc:302] T 00000000000000000000000000000000 P 7c728d90c4154315866d4a69c26d5b15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c728d90c4154315866d4a69c26d5b15; no voters:
I20260927 22:09:22.065243 6044 leader_election.cc:288] T 00000000000000000000000000000000 P 7c728d90c4154315866d4a69c26d5b15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:09:22.065335 6049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c728d90c4154315866d4a69c26d5b15 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:09:22.066061 6049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c728d90c4154315866d4a69c26d5b15 [term 1 LEADER]: Becoming Leader. State: Replica: 7c728d90c4154315866d4a69c26d5b15, State: Running, Role: LEADER
I20260927 22:09:22.066558 6049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c728d90c4154315866d4a69c26d5b15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c728d90c4154315866d4a69c26d5b15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36765 } }
I20260927 22:09:22.067473 6044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c728d90c4154315866d4a69c26d5b15 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:09:22.069496 6050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c728d90c4154315866d4a69c26d5b15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c728d90c4154315866d4a69c26d5b15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c728d90c4154315866d4a69c26d5b15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36765 } } }
I20260927 22:09:22.069859 6050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c728d90c4154315866d4a69c26d5b15 [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:22.070501 6051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c728d90c4154315866d4a69c26d5b15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c728d90c4154315866d4a69c26d5b15. Latest consensus state: current_term: 1 leader_uuid: "7c728d90c4154315866d4a69c26d5b15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c728d90c4154315866d4a69c26d5b15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36765 } } }
I20260927 22:09:22.070883 6051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c728d90c4154315866d4a69c26d5b15 [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:22.072216 6056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:09:22.077131 6056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:09:22.086401 6056 catalog_manager.cc:1347] Generated new cluster ID: 7cece52a0a9044c0ad2eca9c078ed2e5
I20260927 22:09:22.086508 6056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:09:22.098773 6056 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:09:22.099969 6056 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:09:22.112309 6056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c728d90c4154315866d4a69c26d5b15: Generated new TSK 0
I20260927 22:09:22.113137 6056 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:09:22.172435 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36765
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:22.508109 6070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:22.508436 6070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:22.508610 6070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:22.515779 6070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:22.515925 6070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:22.516108 6070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:09:22.525625 6070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:22.526935 6070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:22.528230 6070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:22.541047 6076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:22.541165 6075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:22.541092 6078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:22.542150 6070 server_base.cc:1034] running on GCE node
I20260927 22:09:22.542901 6070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:22.543826 6070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:22.545061 6070 hybrid_clock.cc:648] HybridClock initialized: now 1790546962545014 us; error 41 us; skew 500 ppm
I20260927 22:09:22.545420 6070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:22 dist-test-slave-mp03 krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:22.553114 6070 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:09:22.554965 6070 webserver.cc:460] Webserver started at http://127.0.97.1:42625/ using document root <none> and password file <none>
I20260927 22:09:22.555670 6070 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:22.555814 6070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:22.556114 6070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:22.559398 6070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "146ca7e46d4340f6af418a87952888eb"
format_stamp: "Formatted at 2026-09-27 22:09:22 on dist-test-slave-mp03"
server_key: "d4b838c70442597f37a768787e17bade"
server_key_iv: "555c9ed34f11aaeb7881cc4dff9dd68c"
server_key_version: "encryptionkey@0"
I20260927 22:09:22.560261 6070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "146ca7e46d4340f6af418a87952888eb"
format_stamp: "Formatted at 2026-09-27 22:09:22 on dist-test-slave-mp03"
server_key: "d4b838c70442597f37a768787e17bade"
server_key_iv: "555c9ed34f11aaeb7881cc4dff9dd68c"
server_key_version: "encryptionkey@0"
I20260927 22:09:22.565814 6070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:09:22.569720 6085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:22.571133 6070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:09:22.571321 6070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "146ca7e46d4340f6af418a87952888eb"
format_stamp: "Formatted at 2026-09-27 22:09:22 on dist-test-slave-mp03"
server_key: "d4b838c70442597f37a768787e17bade"
server_key_iv: "555c9ed34f11aaeb7881cc4dff9dd68c"
server_key_version: "encryptionkey@0"
I20260927 22:09:22.571565 6070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:22.586313 6070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:22.590557 6070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:22.590850 6070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:22.591717 6070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:22.592988 6070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:22.593057 6070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:22.593139 6070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:22.593183 6070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:22.624971 6070 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44795
I20260927 22:09:22.625005 6197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44795 every 8 connection(s)
I20260927 22:09:22.626315 6070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:09:22.635162 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 6070
I20260927 22:09:22.635389 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:09:22.636001 388 external_mini_cluster.cc:1428] Setting key fe9212ed2e6873551d8d4252543d90f4
Sep 27 22:09:22 dist-test-slave-mp03 krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:09:22.646570 6200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:22.628900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38347 (local address 127.0.97.62:36765)
0927 22:09:22.629353 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:22.629358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:22.630208 (+ 850us) server_negotiation.cc:407] Connection header received
0927 22:09:22.632000 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:22.632006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:22.632093 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:22.632208 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:22.633675 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:22.634409 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:22.636111 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:22.636506 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:22.636647 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:22.640711 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:22.640716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:22.640720 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:22.640749 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:22.643210 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:22.644102 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:22.644107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:22.644112 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:22.644258 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:22.644640 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:22.644648 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:22.644652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:22.645030 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:22.645174 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:22.645827 (+ 653us) server_negotiation.cc:299] Negotiation successful
0927 22:09:22.646075 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":240,"threads_started":1}
I20260927 22:09:22.647229 6201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:22.629356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36765 (local address 127.0.97.1:38347)
0927 22:09:22.630049 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:22.630086 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:22.631724 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:22.632450 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:22.632459 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:22.632953 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:22.633510 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:22.633527 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:22.634581 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:22.634585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:22.635972 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:22.635984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:22.636757 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:22.636770 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:22.637008 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:22.637747 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:22.637780 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:22.640504 (+ 2724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:22.643348 (+ 2844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:22.643359 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:22.643379 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:22.643975 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:22.644361 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:22.644365 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:22.644367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:22.644531 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:22.645144 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:22.645150 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:22.645461 (+ 311us) client_negotiation.cc:769] Sending connection context
0927 22:09:22.645677 (+ 216us) client_negotiation.cc:240] Negotiation successful
0927 22:09:22.646075 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":282,"threads_started":1}
I20260927 22:09:22.648782 6198 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36765
I20260927 22:09:22.649101 6198 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:22.649962 6198 heartbeater.cc:511] Master 127.0.97.62:36765 requested a full tablet report, sending...
I20260927 22:09:22.652082 6009 ts_manager.cc:194] Registered new tserver with Master: 146ca7e46d4340f6af418a87952888eb (127.0.97.1:44795)
I20260927 22:09:22.653559 6009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38347
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:09:22.686100 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36765
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:23.012300 6204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:23.012589 6204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:23.012748 6204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:23.020032 6204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:23.020146 6204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:23.020355 6204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:09:23.030331 6204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:23.031668 6204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:23.032970 6204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:23.050868 6210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:23.050906 6212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:23.050882 6209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:23.051684 6204 server_base.cc:1034] running on GCE node
I20260927 22:09:23.052349 6204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:23.053363 6204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:23.054631 6204 hybrid_clock.cc:648] HybridClock initialized: now 1790546963054549 us; error 44 us; skew 500 ppm
I20260927 22:09:23.054970 6204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:23 dist-test-slave-mp03 krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:23.058274 6204 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:09:23.059718 6204 webserver.cc:460] Webserver started at http://127.0.97.2:45407/ using document root <none> and password file <none>
I20260927 22:09:23.060362 6204 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:23.060461 6204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:23.060721 6204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:23.063792 6204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "618fb266225b420296773d0c348acdcf"
format_stamp: "Formatted at 2026-09-27 22:09:23 on dist-test-slave-mp03"
server_key: "c5ba228d7563c5eb32d7b7aa2c460247"
server_key_iv: "9c620a16dd4c18b1f2e5a34395c1c9be"
server_key_version: "encryptionkey@0"
I20260927 22:09:23.064508 6204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "618fb266225b420296773d0c348acdcf"
format_stamp: "Formatted at 2026-09-27 22:09:23 on dist-test-slave-mp03"
server_key: "c5ba228d7563c5eb32d7b7aa2c460247"
server_key_iv: "9c620a16dd4c18b1f2e5a34395c1c9be"
server_key_version: "encryptionkey@0"
I20260927 22:09:23.070030 6204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:09:23.073431 6219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:23.074992 6204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 22:09:23.075174 6204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "618fb266225b420296773d0c348acdcf"
format_stamp: "Formatted at 2026-09-27 22:09:23 on dist-test-slave-mp03"
server_key: "c5ba228d7563c5eb32d7b7aa2c460247"
server_key_iv: "9c620a16dd4c18b1f2e5a34395c1c9be"
server_key_version: "encryptionkey@0"
I20260927 22:09:23.075428 6204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:23.088115 6204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:23.091945 6204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:23.092232 6204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:23.093134 6204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:23.094502 6204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:23.094571 6204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:23.094767 6204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:23.094858 6204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:23.125413 6204 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45365
I20260927 22:09:23.125450 6331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45365 every 8 connection(s)
I20260927 22:09:23.126688 6204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:09:23.127113 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 6204
I20260927 22:09:23.127350 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:09:23.127877 388 external_mini_cluster.cc:1428] Setting key ef9008a75f49efc118fd9d80066c286d
Sep 27 22:09:23 dist-test-slave-mp03 krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:09:23.149919 6200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:23.129592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48125 (local address 127.0.97.62:36765)
0927 22:09:23.129787 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:23.129794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:23.131313 (+ 1519us) server_negotiation.cc:407] Connection header received
0927 22:09:23.134127 (+ 2814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:23.134131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:23.134205 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:23.134290 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:23.136471 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:23.137137 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:23.139369 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:23.139772 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:23.139871 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:23.144032 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:23.144037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:23.144040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:23.144070 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:23.146684 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:23.147380 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:23.147387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:23.147393 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:23.147558 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:23.148085 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:23.148092 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:23.148097 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:23.148464 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:23.148610 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:23.149263 (+ 653us) server_negotiation.cc:299] Negotiation successful
0927 22:09:23.149502 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:09:23.150676 6334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:23.130279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36765 (local address 127.0.97.2:48125)
0927 22:09:23.131140 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:23.131190 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:23.133831 (+ 2641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:23.134571 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:23.134584 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:23.135409 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:23.136276 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:23.136305 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:23.137314 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:23.137324 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:23.139213 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:23.139235 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:23.139997 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:23.140006 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:23.140334 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:23.141071 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:23.141104 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:23.143797 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:23.146811 (+ 3014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:23.146817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:23.146831 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:23.147239 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:23.147696 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:23.147700 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:23.147703 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:23.147953 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:23.148583 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:23.148589 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:23.148888 (+ 299us) client_negotiation.cc:769] Sending connection context
0927 22:09:23.149103 (+ 215us) client_negotiation.cc:240] Negotiation successful
0927 22:09:23.149475 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":350,"threads_started":1}
I20260927 22:09:23.152208 6332 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36765
I20260927 22:09:23.152558 6332 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 22:09:23.153455 6332 heartbeater.cc:511] Master 127.0.97.62:36765 requested a full tablet report, sending...
I20260927 22:09:23.155042 6009 ts_manager.cc:194] Registered new tserver with Master: 618fb266225b420296773d0c348acdcf (127.0.97.2:45365)
I20260927 22:09:23.156284 6009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:09:23.181773 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36765
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:23.504042 6337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:23.504321 6337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:23.504513 6337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:23.511801 6337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:23.511945 6337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:23.512131 6337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:09:23.521967 6337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:23.523648 6337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:23.525151 6337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:23.537997 6345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:23.538089 6342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:23.538089 6343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:23.538940 6337 server_base.cc:1034] running on GCE node
I20260927 22:09:23.539549 6337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:23.540484 6337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:23.541718 6337 hybrid_clock.cc:648] HybridClock initialized: now 1790546963541682 us; error 57 us; skew 500 ppm
I20260927 22:09:23.542027 6337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:23 dist-test-slave-mp03 krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:23.545128 6337 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:09:23.546573 6337 webserver.cc:460] Webserver started at http://127.0.97.3:40085/ using document root <none> and password file <none>
I20260927 22:09:23.547300 6337 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:23.547442 6337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:23.547748 6337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:23.550709 6337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "16dc4bb6e6c04d0b994731e5bd8305a7"
format_stamp: "Formatted at 2026-09-27 22:09:23 on dist-test-slave-mp03"
server_key: "d872f4ea0b0fe34692d2a5324c466a60"
server_key_iv: "1b1c168beaade06241148e72f1ead289"
server_key_version: "encryptionkey@0"
I20260927 22:09:23.551468 6337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "16dc4bb6e6c04d0b994731e5bd8305a7"
format_stamp: "Formatted at 2026-09-27 22:09:23 on dist-test-slave-mp03"
server_key: "d872f4ea0b0fe34692d2a5324c466a60"
server_key_iv: "1b1c168beaade06241148e72f1ead289"
server_key_version: "encryptionkey@0"
I20260927 22:09:23.556378 6337 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260927 22:09:23.559943 6352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:23.561223 6337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:09:23.561391 6337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "16dc4bb6e6c04d0b994731e5bd8305a7"
format_stamp: "Formatted at 2026-09-27 22:09:23 on dist-test-slave-mp03"
server_key: "d872f4ea0b0fe34692d2a5324c466a60"
server_key_iv: "1b1c168beaade06241148e72f1ead289"
server_key_version: "encryptionkey@0"
I20260927 22:09:23.561599 6337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:23.576519 6337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:23.580847 6337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:23.581144 6337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:23.582366 6337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:23.583866 6337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:23.583979 6337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:23.584136 6337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:23.584211 6337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:23.615810 6337 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37127
I20260927 22:09:23.615844 6464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37127 every 8 connection(s)
I20260927 22:09:23.617146 6337 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:09:23.620246 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 6337
I20260927 22:09:23.620419 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:09:23.621011 388 external_mini_cluster.cc:1428] Setting key f258dec02125c96cb8f88f18666c404a
Sep 27 22:09:23 dist-test-slave-mp03 krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:09:23.638825 6200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:23.619773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49713 (local address 127.0.97.62:36765)
0927 22:09:23.619959 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:23.619964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:23.621334 (+ 1370us) server_negotiation.cc:407] Connection header received
0927 22:09:23.623715 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:23.623721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:23.623802 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:23.623875 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:23.626067 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:23.626867 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:23.629138 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:23.629539 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:23.629652 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:23.633100 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:23.633105 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:23.633108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:23.633139 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:23.636042 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:23.636889 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:23.636894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:23.636896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:23.637017 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:23.637376 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:23.637378 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:23.637380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:23.637635 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:23.637738 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:23.638361 (+ 623us) server_negotiation.cc:299] Negotiation successful
0927 22:09:23.638527 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:09:23.639767 6467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:23.620347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36765 (local address 127.0.97.3:49713)
0927 22:09:23.621186 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:23.621230 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:23.623416 (+ 2186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:23.624354 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:23.624365 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:23.625067 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:23.625863 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:23.625888 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:23.627011 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:23.627017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:23.628945 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:23.628959 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:23.629758 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:23.629765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:23.630044 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:23.630801 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:23.630823 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:23.632911 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:23.636285 (+ 3374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:23.636294 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:23.636312 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:23.636788 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:23.637108 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:23.637111 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:23.637113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:23.637287 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:23.637734 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:23.637739 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:23.638053 (+ 314us) client_negotiation.cc:769] Sending connection context
0927 22:09:23.638275 (+ 222us) client_negotiation.cc:240] Negotiation successful
0927 22:09:23.638736 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":338,"threads_started":1}
I20260927 22:09:23.641053 6465 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36765
I20260927 22:09:23.641356 6465 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:23.642057 6465 heartbeater.cc:511] Master 127.0.97.62:36765 requested a full tablet report, sending...
I20260927 22:09:23.643484 6009 ts_manager.cc:194] Registered new tserver with Master: 16dc4bb6e6c04d0b994731e5bd8305a7 (127.0.97.3:37127)
I20260927 22:09:23.644241 6009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49713
I20260927 22:09:23.648506 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:09:23.657188 6198 heartbeater.cc:503] Master 127.0.97.62:36765 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/test-admin.keytab.
Sep 27 22:09:23 dist-test-slave-mp03 krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:23.676537 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 22:09:23.681958 6472 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:23.679447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57516 (local address 127.0.97.1:44795)
0927 22:09:23.680079 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:23.680086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:23.680112 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:09:23.680171 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:23.680177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:23.680505 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:23.680631 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:23.681527 (+ 896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57516: BlockingRecv error: recv got EOF from 127.0.0.1:57516 (error 108)
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":298,"threads_started":1}
W20260927 22:09:23.682127 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/krb5cc' of type FILE
Sep 27 22:09:23 dist-test-slave-mp03 krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:23.683741 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 22:09:23 dist-test-slave-mp03 krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:23.685086 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 22:09:23 dist-test-slave-mp03 krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:09:23.699718 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:23.686824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57520 (local address 127.0.97.1:44795)
0927 22:09:23.687022 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:23.687033 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:23.687164 (+ 131us) server_negotiation.cc:407] Connection header received
0927 22:09:23.687287 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:23.687291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:23.687343 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:23.687421 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:23.688318 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:23.689258 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:23.690426 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:23.691019 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:23.691112 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:23.694526 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:23.694538 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:23.694552 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:23.694582 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:23.697411 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:23.697985 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:23.697992 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:23.697994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:23.698104 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:23.698404 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:23.698407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:23.698409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:23.698802 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:23.699023 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:23.699280 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 22:09:23.699468 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 22:09:23.705251 6472 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:23.703668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57534 (local address 127.0.97.1:44795)
0927 22:09:23.703816 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:23.703820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:23.704165 (+ 345us) server_negotiation.cc:407] Connection header received
0927 22:09:23.704274 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:23.704278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:23.704341 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:23.704417 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:23.704990 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57534: BlockingRecv error: recv got EOF from 127.0.0.1:57534 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 22:09:23.705417 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/krb5cc' of type FILE
Sep 27 22:09:23 dist-test-slave-mp03 krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:23.706915 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 22:09:23 dist-test-slave-mp03 krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:23.708155 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 22:09:23 dist-test-slave-mp03 krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:09:23.721674 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:23.709342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57546 (local address 127.0.97.1:44795)
0927 22:09:23.709529 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:23.709537 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:23.709815 (+ 278us) server_negotiation.cc:407] Connection header received
0927 22:09:23.710002 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:23.710007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:23.710058 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:23.710152 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:23.711086 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:23.711796 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:23.712973 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:23.713412 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:23.713539 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:23.716828 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:23.716833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:23.716836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:23.716871 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:23.719358 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:23.719954 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:23.719960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:23.719964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:23.720086 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:23.720435 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:23.720442 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:23.720446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:23.720695 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:23.720965 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:23.721219 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 22:09:23.721407 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 22:09:23.725538 6472 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:23.723895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57562 (local address 127.0.97.1:44795)
0927 22:09:23.724035 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:23.724039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:23.724370 (+ 331us) server_negotiation.cc:407] Connection header received
0927 22:09:23.724490 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:23.724496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:23.724570 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:23.724642 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:23.725255 (+ 613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57562: BlockingRecv error: recv got EOF from 127.0.0.1:57562 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 22:09:23.725701 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546912983837-388-0/krb5kdc/krb5cc' of type FILE
Sep 27 22:09:23 dist-test-slave-mp03 krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:23.727229 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 22:09:23 dist-test-slave-mp03 krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:23.728539 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 22:09:23 dist-test-slave-mp03 krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:09:23.741920 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:23.729821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57572 (local address 127.0.97.1:44795)
0927 22:09:23.730034 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:23.730040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:23.730178 (+ 138us) server_negotiation.cc:407] Connection header received
0927 22:09:23.730270 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:23.730273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:23.730325 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:23.730401 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:23.731480 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:23.732462 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:23.733583 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:23.733965 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:23.734051 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:23.737267 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:23.737277 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:23.737283 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:23.737324 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:23.739756 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:23.740306 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:23.740310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:23.740312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:23.740467 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:23.740764 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:23.740768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:23.740770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:23.741048 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:23.741281 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:23.741480 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 22:09:23.741640 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260927 22:09:23.743511 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 6070
I20260927 22:09:23.754052 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:23.744172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57574 (local address 127.0.97.1:44795)
0927 22:09:23.744313 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:23.744317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:23.744617 (+ 300us) server_negotiation.cc:407] Connection header received
0927 22:09:23.744716 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:23.744719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:23.744767 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:23.744841 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:23.745686 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:23.746316 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:23.747490 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:23.747905 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:23.748002 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:23.749834 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:23.749840 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:23.749845 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:23.749873 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:23.752096 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:23.752637 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:23.752641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:23.752643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:23.752768 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:23.753052 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:23.753055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:23.753057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:23.753273 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:23.753454 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:23.753652 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 22:09:23.753800 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 22:09:23.754673 6193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:23.838330 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 6070
I20260927 22:09:23.854379 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 6204
Sep 27 22:09:23 dist-test-slave-mp03 krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:09:23.869236 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:23.855253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37422 (local address 127.0.97.2:45365)
0927 22:09:23.855774 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:23.855781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:23.855805 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:09:23.855878 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:23.855884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:23.856084 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:23.856203 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:23.857120 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:23.858111 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:23.859489 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:23.860046 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:23.860143 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:23.863950 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:23.863961 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:23.863975 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:23.864005 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:23.866808 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:23.867448 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:23.867454 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:23.867457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:23.867568 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:23.867883 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:23.867886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:23.867888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:23.868243 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:23.868490 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:23.868694 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 22:09:23.868918 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":279,"threads_started":1}
I20260927 22:09:23.870687 6327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:23.957901 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 6204
I20260927 22:09:23.973874 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 6337
Sep 27 22:09:23 dist-test-slave-mp03 krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:09:23.989491 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:23.974900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59362 (local address 127.0.97.3:37127)
0927 22:09:23.975519 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:23.975527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:23.975556 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:09:23.975613 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:23.975619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:23.975835 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:23.976027 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:23.977139 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:23.978133 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:23.979463 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:23.980020 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:23.980161 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:23.984566 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:23.984577 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:23.984592 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:23.984625 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:23.987261 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:23.987854 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:23.987861 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:23.987864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:23.987971 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:23.988287 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:23.988290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:23.988292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:23.988604 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:23.988830 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:23.989026 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 22:09:23.989207 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":291,"threads_started":1}
I20260927 22:09:23.990944 6460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:24.072024 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 6337
I20260927 22:09:24.087813 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 5977
I20260927 22:09:24.088888 6039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:24.189560 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 5977
2026-09-27T22:09:24Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3905 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:09:24 dist-test-slave-mp03 krb5kdc[6492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:09:24 dist-test-slave-mp03 krb5kdc[6492](info): set up 2 sockets
Sep 27 22:09:24 dist-test-slave-mp03 krb5kdc[6492](info): commencing operation
krb5kdc: starting...
W20260927 22:09:25.367503 388 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.004s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:09:25 dist-test-slave-mp03 krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:09:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:09:25Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260927 22:09:25.579263 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35323
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:25.895774 6505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:25.896075 6505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:25.896163 6505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:25.903514 6505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:09:25.903633 6505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:25.903678 6505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:25.903753 6505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:09:25.903798 6505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:09:25.913559 6505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:25.914858 6505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:25.916100 6505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:25.927973 6510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:25.927959 6513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:25.927959 6511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:25.928604 6505 server_base.cc:1034] running on GCE node
I20260927 22:09:25.929446 6505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:25.930855 6505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:25.932091 6505 hybrid_clock.cc:648] HybridClock initialized: now 1790546965932043 us; error 38 us; skew 500 ppm
I20260927 22:09:25.932453 6505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:25 dist-test-slave-mp03 krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546965, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:25.935515 6505 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260927 22:09:25.936911 6505 webserver.cc:460] Webserver started at http://127.0.97.62:33957/ using document root <none> and password file <none>
I20260927 22:09:25.937618 6505 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:25.937767 6505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:25.938066 6505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:25.941075 6505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d"
format_stamp: "Formatted at 2026-09-27 22:09:25 on dist-test-slave-mp03"
server_key: "92944a62baccf8d8ac0b6301e94fbdac"
server_key_iv: "e1a397913d7f7eaec9a4eb95d416b7e8"
server_key_version: "encryptionkey@0"
I20260927 22:09:25.941851 6505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d"
format_stamp: "Formatted at 2026-09-27 22:09:25 on dist-test-slave-mp03"
server_key: "92944a62baccf8d8ac0b6301e94fbdac"
server_key_iv: "e1a397913d7f7eaec9a4eb95d416b7e8"
server_key_version: "encryptionkey@0"
I20260927 22:09:25.947137 6505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260927 22:09:25.950570 6520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:25.952044 6505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:09:25.952232 6505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d"
format_stamp: "Formatted at 2026-09-27 22:09:25 on dist-test-slave-mp03"
server_key: "92944a62baccf8d8ac0b6301e94fbdac"
server_key_iv: "e1a397913d7f7eaec9a4eb95d416b7e8"
server_key_version: "encryptionkey@0"
I20260927 22:09:25.952432 6505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:25.978122 6505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:25.982564 6505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:25.982908 6505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:26.000039 6505 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35323
I20260927 22:09:26.000061 6571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35323 every 8 connection(s)
I20260927 22:09:26.001462 6505 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:09:26.005255 6572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:26.008985 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 6505
I20260927 22:09:26.009202 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:09:26.010025 388 external_mini_cluster.cc:1428] Setting key b8be604890e6d2f28621492bc3659786
I20260927 22:09:26.018767 6572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d: Bootstrap starting.
I20260927 22:09:26.022513 6572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:26.023906 6572 log.cc:824] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:09:26 dist-test-slave-mp03 krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:09:26.027091 6572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d: No bootstrap required, opened a new log
I20260927 22:09:26.033842 6575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:26.012231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37414 (local address 127.0.97.62:35323)
0927 22:09:26.012956 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:26.012968 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:26.013008 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:09:26.014684 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:26.014708 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:26.015118 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:26.015459 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:26.017207 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:26.018564 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:26.020297 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:26.021027 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:26.021181 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:26.026223 (+ 5042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:26.026231 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:26.026247 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:26.026281 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:26.029586 (+ 3305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:26.030375 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:26.030382 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:26.030390 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:26.030564 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:26.031009 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:26.031014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:26.031026 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:26.031566 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:26.031882 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:26.032359 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 22:09:26.032798 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":340,"threads_started":1}
I20260927 22:09:26.035048 6572 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } }
I20260927 22:09:26.035527 6572 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:26.035643 6572 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae6791bd85ae4fceb7d3ac0a20c86d2d, State: Initialized, Role: FOLLOWER
I20260927 22:09:26.036284 6572 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } }
I20260927 22:09:26.036579 6572 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:09:26.036732 6572 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:09:26.036934 6572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:26.039225 6572 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } }
I20260927 22:09:26.039685 6572 leader_election.cc:302] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae6791bd85ae4fceb7d3ac0a20c86d2d; no voters:
I20260927 22:09:26.040225 6572 leader_election.cc:288] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:09:26.040369 6577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:09:26.041885 6577 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [term 1 LEADER]: Becoming Leader. State: Replica: ae6791bd85ae4fceb7d3ac0a20c86d2d, State: Running, Role: LEADER
I20260927 22:09:26.042323 6572 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:09:26.042461 6577 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } }
I20260927 22:09:26.045589 6578 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } }
I20260927 22:09:26.045928 6578 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:26.046311 6579 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae6791bd85ae4fceb7d3ac0a20c86d2d. Latest consensus state: current_term: 1 leader_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } }
I20260927 22:09:26.046766 6579 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:26.048790 6588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:09:26.053824 6588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:09:26.062165 6588 catalog_manager.cc:1347] Generated new cluster ID: 0a21bad7669c4c3ea45c68ac34cabb13
I20260927 22:09:26.062340 6588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:09:26.079293 6588 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:09:26.080745 6588 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:09:26.090713 6588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d: Generated new TSK 0
I20260927 22:09:26.091601 6588 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260927 22:09:26.154170 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35323
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:26.482218 6598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:26.482503 6598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:26.482645 6598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:26.490051 6598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:26.490155 6598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:26.490266 6598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:09:26.499993 6598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:26.501197 6598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:26.502426 6598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:26.515154 6604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:26.515228 6603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:26.515261 6606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:26.516237 6598 server_base.cc:1034] running on GCE node
I20260927 22:09:26.516858 6598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:26.517776 6598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:26.518996 6598 hybrid_clock.cc:648] HybridClock initialized: now 1790546966518958 us; error 52 us; skew 500 ppm
I20260927 22:09:26.519330 6598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:26 dist-test-slave-mp03 krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546966, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:26.522390 6598 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 22:09:26.523910 6598 webserver.cc:460] Webserver started at http://127.0.97.1:36345/ using document root <none> and password file <none>
I20260927 22:09:26.524581 6598 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:26.524720 6598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:26.525010 6598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:26.528018 6598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "100d5e2a45074ac29b114b3bf0e3f693"
format_stamp: "Formatted at 2026-09-27 22:09:26 on dist-test-slave-mp03"
server_key: "35346e211cafd6ae54bf5650c0773981"
server_key_iv: "c7265530678a053eda5f55157e49fe4d"
server_key_version: "encryptionkey@0"
I20260927 22:09:26.528792 6598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "100d5e2a45074ac29b114b3bf0e3f693"
format_stamp: "Formatted at 2026-09-27 22:09:26 on dist-test-slave-mp03"
server_key: "35346e211cafd6ae54bf5650c0773981"
server_key_iv: "c7265530678a053eda5f55157e49fe4d"
server_key_version: "encryptionkey@0"
I20260927 22:09:26.533666 6598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:09:26.537035 6613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:26.538309 6598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:09:26.538476 6598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "100d5e2a45074ac29b114b3bf0e3f693"
format_stamp: "Formatted at 2026-09-27 22:09:26 on dist-test-slave-mp03"
server_key: "35346e211cafd6ae54bf5650c0773981"
server_key_iv: "c7265530678a053eda5f55157e49fe4d"
server_key_version: "encryptionkey@0"
I20260927 22:09:26.538720 6598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:26.562644 6598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:26.566583 6598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:26.566915 6598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:26.567821 6598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:26.569073 6598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:26.569145 6598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:26.569303 6598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:26.569361 6598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:26.601713 6598 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43179
I20260927 22:09:26.601771 6725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43179 every 8 connection(s)
I20260927 22:09:26.603071 6598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:09:26.607155 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 6598
I20260927 22:09:26.607358 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:09:26.608011 388 external_mini_cluster.cc:1428] Setting key 1f1e440b3685fc847e957c7aea5d13ab
Sep 27 22:09:26 dist-test-slave-mp03 krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546966, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:09:26.622623 6728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:26.605804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57965 (local address 127.0.97.62:35323)
0927 22:09:26.606249 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:26.606254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:26.607158 (+ 904us) server_negotiation.cc:407] Connection header received
0927 22:09:26.609182 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:26.609188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:26.609276 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:26.609390 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:26.610992 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:26.611704 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:26.613235 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:26.613628 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:26.613761 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:26.617138 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:26.617143 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:26.617146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:26.617176 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:26.619671 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:26.620384 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:26.620397 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:26.620399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:26.620517 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:26.620942 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:26.620953 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:26.620955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:26.621192 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:26.621342 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:26.622026 (+ 684us) server_negotiation.cc:299] Negotiation successful
0927 22:09:26.622240 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":246,"threads_started":1}
I20260927 22:09:26.623605 6729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:26.606253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.97.1:57965)
0927 22:09:26.606995 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:26.607038 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:26.608894 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:26.609648 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:26.609661 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:26.610205 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:26.610836 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:26.610854 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:26.611853 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:26.611858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:26.613074 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:26.613085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:26.613845 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:26.613854 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:26.614085 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:26.614810 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:26.614833 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:26.616920 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:26.619819 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:26.619828 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:26.619845 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:26.620259 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:26.620632 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:26.620638 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:26.620641 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:26.620826 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:26.621315 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:26.621322 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:26.621665 (+ 343us) client_negotiation.cc:769] Sending connection context
0927 22:09:26.621924 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 22:09:26.622456 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":334,"threads_started":1}
I20260927 22:09:26.625231 6726 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35323
I20260927 22:09:26.625573 6726 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:09:26.626488 6726 heartbeater.cc:511] Master 127.0.97.62:35323 requested a full tablet report, sending...
I20260927 22:09:26.628567 6537 ts_manager.cc:194] Registered new tserver with Master: 100d5e2a45074ac29b114b3bf0e3f693 (127.0.97.1:43179)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260927 22:09:26.630378 6537 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260927 22:09:26.653949 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35323
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:26.975988 6732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:26.976289 6732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:26.976384 6732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:26.983696 6732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:26.983808 6732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:26.983928 6732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:09:26.993554 6732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:26.994903 6732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:26.996132 6732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:27.011564 6738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:27.012130 6740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:27.012295 6737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:27.012701 6732 server_base.cc:1034] running on GCE node
I20260927 22:09:27.013424 6732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:27.014513 6732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:27.015767 6732 hybrid_clock.cc:648] HybridClock initialized: now 1790546967015694 us; error 68 us; skew 500 ppm
I20260927 22:09:27.016115 6732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:27 dist-test-slave-mp03 krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546967, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:27.019371 6732 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 22:09:27.020668 6732 webserver.cc:460] Webserver started at http://127.0.97.2:43067/ using document root <none> and password file <none>
I20260927 22:09:27.021274 6732 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:27.021588 6732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:27.021899 6732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:27.024798 6732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "ebdd20bf62754f138e1d99f4509ba2bd"
format_stamp: "Formatted at 2026-09-27 22:09:27 on dist-test-slave-mp03"
server_key: "4847c3d6318553c4f341b31b4b071b0c"
server_key_iv: "dadc3ec9d72c794ed871564e2571cf75"
server_key_version: "encryptionkey@0"
I20260927 22:09:27.025538 6732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ebdd20bf62754f138e1d99f4509ba2bd"
format_stamp: "Formatted at 2026-09-27 22:09:27 on dist-test-slave-mp03"
server_key: "4847c3d6318553c4f341b31b4b071b0c"
server_key_iv: "dadc3ec9d72c794ed871564e2571cf75"
server_key_version: "encryptionkey@0"
I20260927 22:09:27.031210 6732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:09:27.034904 6747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:27.036209 6732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:09:27.036427 6732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "ebdd20bf62754f138e1d99f4509ba2bd"
format_stamp: "Formatted at 2026-09-27 22:09:27 on dist-test-slave-mp03"
server_key: "4847c3d6318553c4f341b31b4b071b0c"
server_key_iv: "dadc3ec9d72c794ed871564e2571cf75"
server_key_version: "encryptionkey@0"
I20260927 22:09:27.036603 6732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:27.057783 6732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:27.061771 6732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:27.062060 6732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:27.063058 6732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:27.064317 6732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:27.064390 6732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:27.064553 6732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:27.064603 6732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:27.101914 6732 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38917
I20260927 22:09:27.101971 6859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38917 every 8 connection(s)
I20260927 22:09:27.103363 6732 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:09:27.113721 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 6732
I20260927 22:09:27.113951 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:09:27.114521 388 external_mini_cluster.cc:1428] Setting key 626de9fc1baf79eed96b9931612d3126
Sep 27 22:09:27 dist-test-slave-mp03 krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546967, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:09:27.124879 6728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:27.106132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52767 (local address 127.0.97.62:35323)
0927 22:09:27.106356 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:27.106362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:27.107590 (+ 1228us) server_negotiation.cc:407] Connection header received
0927 22:09:27.109461 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:27.109465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:27.109533 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:27.109623 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:27.111264 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:27.111960 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:27.113553 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:27.114197 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:27.114370 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:27.118094 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:27.118104 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:27.118112 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:27.118158 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:27.121714 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:27.122416 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:27.122424 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:27.122430 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:27.122578 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:27.123173 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:27.123180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:27.123187 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:27.123467 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:27.123619 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:27.124321 (+ 702us) server_negotiation.cc:299] Negotiation successful
0927 22:09:27.124530 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 22:09:27.125881 6862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:27.106708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.97.2:52767)
0927 22:09:27.107426 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:27.107470 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:27.109196 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:27.109899 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:27.109908 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:27.110435 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:27.111083 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:27.111101 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:27.112106 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:27.112110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:27.113389 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:27.113400 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:27.114427 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:27.114432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:27.114733 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:27.115498 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:27.115530 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:27.117869 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:27.121856 (+ 3987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:27.121862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:27.121876 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:27.122288 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:27.122759 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:27.122763 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:27.122765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:27.123045 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:27.123593 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:27.123602 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:27.123960 (+ 358us) client_negotiation.cc:769] Sending connection context
0927 22:09:27.124217 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 22:09:27.124757 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":277,"threads_started":1}
I20260927 22:09:27.127352 6860 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35323
I20260927 22:09:27.127635 6860 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:27.128306 6860 heartbeater.cc:511] Master 127.0.97.62:35323 requested a full tablet report, sending...
I20260927 22:09:27.129732 6537 ts_manager.cc:194] Registered new tserver with Master: ebdd20bf62754f138e1d99f4509ba2bd (127.0.97.2:38917)
I20260927 22:09:27.131014 6537 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52767
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260927 22:09:27.161550 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35323
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:27.491065 6865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:27.491345 6865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:27.491436 6865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:27.498410 6865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:27.498513 6865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:27.498697 6865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:09:27.508258 6865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:27.509439 6865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:27.510828 6865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:27.523592 6870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:27.523627 6873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:27.523535 6871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:27.524503 6865 server_base.cc:1034] running on GCE node
I20260927 22:09:27.525255 6865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:27.526177 6865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:27.527361 6865 hybrid_clock.cc:648] HybridClock initialized: now 1790546967527299 us; error 57 us; skew 500 ppm
I20260927 22:09:27.527679 6865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:27 dist-test-slave-mp03 krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546967, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:27.530808 6865 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 22:09:27.532207 6865 webserver.cc:460] Webserver started at http://127.0.97.3:32909/ using document root <none> and password file <none>
I20260927 22:09:27.532840 6865 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:27.532938 6865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:27.533200 6865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:27.536144 6865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "0f99af01ae61446fb3569c69c2e81936"
format_stamp: "Formatted at 2026-09-27 22:09:27 on dist-test-slave-mp03"
server_key: "2fdcb8a665969ad275db30add421d178"
server_key_iv: "b774dadebfe4685f3d3d016773824c4b"
server_key_version: "encryptionkey@0"
I20260927 22:09:27.536872 6865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0f99af01ae61446fb3569c69c2e81936"
format_stamp: "Formatted at 2026-09-27 22:09:27 on dist-test-slave-mp03"
server_key: "2fdcb8a665969ad275db30add421d178"
server_key_iv: "b774dadebfe4685f3d3d016773824c4b"
server_key_version: "encryptionkey@0"
I20260927 22:09:27.541987 6865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:09:27.545657 6880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:27.547101 6865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 22:09:27.547256 6865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "0f99af01ae61446fb3569c69c2e81936"
format_stamp: "Formatted at 2026-09-27 22:09:27 on dist-test-slave-mp03"
server_key: "2fdcb8a665969ad275db30add421d178"
server_key_iv: "b774dadebfe4685f3d3d016773824c4b"
server_key_version: "encryptionkey@0"
I20260927 22:09:27.547452 6865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:27.582702 6865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:27.587229 6865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:27.587528 6865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:27.588507 6865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:27.589808 6865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:27.589879 6865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:27.590044 6865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:27.590102 6865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:27.623180 6865 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38105
I20260927 22:09:27.623224 6992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38105 every 8 connection(s)
I20260927 22:09:27.624434 6865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:09:27.632301 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 6865
I20260927 22:09:27.632468 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:09:27.632974 388 external_mini_cluster.cc:1428] Setting key 05f6928c4fbcb0f85ff11a87fe0bfb52
I20260927 22:09:27.633910 6726 heartbeater.cc:503] Master 127.0.97.62:35323 was elected leader, sending a full tablet report...
Sep 27 22:09:27 dist-test-slave-mp03 krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546967, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:09:27.644623 6995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:27.627010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49187 (local address 127.0.97.62:35323)
0927 22:09:27.627427 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:27.627432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:27.628354 (+ 922us) server_negotiation.cc:407] Connection header received
0927 22:09:27.630158 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:27.630163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:27.630250 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:27.630366 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:27.631847 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:27.633080 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:27.635177 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:27.635736 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:27.635875 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:27.639229 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:27.639233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:27.639236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:27.639265 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:27.641773 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:27.642514 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:27.642524 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:27.642528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:27.642689 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:27.643128 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:27.643132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:27.643134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:27.643384 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:27.643504 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:27.644138 (+ 634us) server_negotiation.cc:299] Negotiation successful
0927 22:09:27.644282 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":235,"threads_started":1}
I20260927 22:09:27.645642 6996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:27.627527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.97.3:49187)
0927 22:09:27.628197 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:27.628239 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:27.629894 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:27.630639 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:27.630655 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:27.631149 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:27.631685 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:27.631703 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:27.633218 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:27.633222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:27.635007 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:27.635022 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:27.635959 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:27.635966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:27.636208 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:27.636876 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:27.636897 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:27.639024 (+ 2127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:27.641904 (+ 2880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:27.641913 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:27.641929 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:27.642387 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:27.642821 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:27.642824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:27.642827 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:27.643002 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:27.643484 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:27.643489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:27.643783 (+ 294us) client_negotiation.cc:769] Sending connection context
0927 22:09:27.644057 (+ 274us) client_negotiation.cc:240] Negotiation successful
0927 22:09:27.644448 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":274,"threads_started":1}
I20260927 22:09:27.647037 6993 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35323
I20260927 22:09:27.647341 6993 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:27.648051 6993 heartbeater.cc:511] Master 127.0.97.62:35323 requested a full tablet report, sending...
I20260927 22:09:27.649482 6536 ts_manager.cc:194] Registered new tserver with Master: 0f99af01ae61446fb3569c69c2e81936 (127.0.97.3:38105)
I20260927 22:09:27.650331 6536 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49187
I20260927 22:09:27.660353 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260927 22:09:27.706863 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39403
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35323,127.0.97.61:39403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:28.024518 6999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:28.024808 6999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:28.024984 6999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:28.032582 6999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:09:28.032702 6999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:28.032786 6999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:28.032866 6999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:09:28.032933 6999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:09:28.043689 6999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35323,127.0.97.61:39403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
W20260927 22:09:28.044094 6999 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:35323,127.0.97.61:39403'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 22:09:28.045336 6999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:28.046765 6999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:28.059183 7005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:28.059172 7004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:28.059194 7007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:28.060427 6999 server_base.cc:1034] running on GCE node
I20260927 22:09:28.061209 6999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:28.062732 6999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:28.063982 6999 hybrid_clock.cc:648] HybridClock initialized: now 1790546968063893 us; error 57 us; skew 500 ppm
I20260927 22:09:28.064400 6999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:28 dist-test-slave-mp03 krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:28.067878 6999 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260927 22:09:28.069397 6999 webserver.cc:460] Webserver started at http://127.0.97.61:39835/ using document root <none> and password file <none>
I20260927 22:09:28.070143 6999 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:28.070288 6999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:28.070680 6999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:28.073814 6999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/data/instance:
uuid: "ef79440c0b4c4fba8b6433b957ca03c3"
format_stamp: "Formatted at 2026-09-27 22:09:28 on dist-test-slave-mp03"
server_key: "e350ff6d53bdd74a6daacc5b6cc5fc1c"
server_key_iv: "bfae5644e7a994eaf51fe64c7bff2d5b"
server_key_version: "encryptionkey@0"
I20260927 22:09:28.074798 6999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/wal/instance:
uuid: "ef79440c0b4c4fba8b6433b957ca03c3"
format_stamp: "Formatted at 2026-09-27 22:09:28 on dist-test-slave-mp03"
server_key: "e350ff6d53bdd74a6daacc5b6cc5fc1c"
server_key_iv: "bfae5644e7a994eaf51fe64c7bff2d5b"
server_key_version: "encryptionkey@0"
I20260927 22:09:28.080389 6999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:09:28.083956 7014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:28.085511 6999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:09:28.085696 6999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/wal
uuid: "ef79440c0b4c4fba8b6433b957ca03c3"
format_stamp: "Formatted at 2026-09-27 22:09:28 on dist-test-slave-mp03"
server_key: "e350ff6d53bdd74a6daacc5b6cc5fc1c"
server_key_iv: "bfae5644e7a994eaf51fe64c7bff2d5b"
server_key_version: "encryptionkey@0"
I20260927 22:09:28.085924 6999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:28.119159 6999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:28.123356 6999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:28.123628 6999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:28.133611 6860 heartbeater.cc:503] Master 127.0.97.62:35323 was elected leader, sending a full tablet report...
I20260927 22:09:28.141569 6999 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:39403
I20260927 22:09:28.141607 7065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:39403 every 8 connection(s)
I20260927 22:09:28.143177 6999 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/data/info.pb
I20260927 22:09:28.146512 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 6999
I20260927 22:09:28.147755 7066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:28.158979 7066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:09:28 dist-test-slave-mp03 krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 27 22:09:28 dist-test-slave-mp03 krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:09:28.173619 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.148695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45394 (local address 127.0.97.61:39403)
0927 22:09:28.149537 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:28.149556 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:28.149611 (+ 55us) server_negotiation.cc:407] Connection header received
0927 22:09:28.151751 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:28.151782 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:28.152359 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:28.152858 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:28.154190 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.156576 (+ 2386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.158070 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.159291 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.159506 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.164926 (+ 5420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:28.164934 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:28.164946 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:28.164983 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:28.168441 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.169460 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.169480 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.169488 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.169711 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.170225 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.170232 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.170236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.171137 (+ 901us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:28.171483 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:28.172037 (+ 554us) server_negotiation.cc:299] Negotiation successful
0927 22:09:28.172486 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":318,"threads_started":1}
I20260927 22:09:28.177716 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.162471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.0.1:37420)
0927 22:09:28.163658 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:28.163706 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:28.163867 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:28.164356 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:28.164361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:28.164977 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:28.165281 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.165291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.166447 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.166459 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.167713 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.167725 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.168448 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.168455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.168633 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.169284 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:28.169301 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:28.171804 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:28.174956 (+ 3152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:28.174967 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:28.174975 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:28.175578 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:28.176016 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:28.176020 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:28.176024 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:28.176169 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:28.176957 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:28.176962 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:28.177096 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 22:09:28.177241 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 22:09:28.177459 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"thread_start_us":820,"threads_started":1}
I20260927 22:09:28.178040 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.162487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37420 (local address 127.0.97.62:35323)
0927 22:09:28.163061 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:28.163069 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:28.163814 (+ 745us) server_negotiation.cc:407] Connection header received
0927 22:09:28.164007 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:28.164017 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:28.164130 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:28.164276 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:28.165430 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.166310 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.167860 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.168265 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.168408 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.171998 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:28.172003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:28.172006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:28.172037 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:28.174824 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.175759 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.175764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.175766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.175900 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.176340 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.176344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.176346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.176837 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:28.176948 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:28.177383 (+ 435us) server_negotiation.cc:299] Negotiation successful
0927 22:09:28.177573 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":277,"threads_started":1}
I20260927 22:09:28.179822 7066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:09:28 dist-test-slave-mp03 krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 22:09:28.192570 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.180403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39403 (local address 127.0.0.1:45400)
0927 22:09:28.180602 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:28.180617 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:28.180689 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:28.181073 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:28.181077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:28.181415 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:28.181531 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.181538 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.182453 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.182460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.183376 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.183387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.184034 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.184039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.184263 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.184802 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:28.184818 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:28.186932 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:28.189911 (+ 2979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:28.189918 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:28.189923 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:28.190383 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:28.190854 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:28.190859 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:28.190863 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:28.191000 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:28.191526 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:28.191531 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:28.191688 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 22:09:28.191836 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 22:09:28.192141 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":6}
I20260927 22:09:28.192728 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.180638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45400 (local address 127.0.97.61:39403)
0927 22:09:28.180866 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:28.180870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:28.180882 (+ 12us) server_negotiation.cc:407] Connection header received
0927 22:09:28.180914 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:28.180917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:28.180960 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:28.181060 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:28.181653 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.182335 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.183505 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.183896 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.184025 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.187137 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:28.187142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:28.187146 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:28.187178 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:28.189778 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.190500 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.190510 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.190513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.190729 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.191108 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.191116 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.191119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.191402 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:28.191551 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:28.191910 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 22:09:28.192286 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":199,"server-negotiator.queue_time_us":97}
I20260927 22:09:28.197474 7066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3: Bootstrap starting.
I20260927 22:09:28.200529 7066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:28.201493 7066 log.cc:824] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:28.204286 7066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3: No bootstrap required, opened a new log
I20260927 22:09:28.211225 7066 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } }
I20260927 22:09:28.211526 7066 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:28.211598 7066 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef79440c0b4c4fba8b6433b957ca03c3, State: Initialized, Role: FOLLOWER
I20260927 22:09:28.212106 7066 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } }
I20260927 22:09:28.213063 7073 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } } }
I20260927 22:09:28.213631 7073 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:09:28.214254 7066 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 22:09:28.222903 7084 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:09:28.223047 7084 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:09:28.224786 6999 master_runner.cc:428] Detected that this master ef79440c0b4c4fba8b6433b957ca03c3 is joining an existing cluster
I20260927 22:09:28.224890 6999 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:39403
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260927 22:09:28.252032 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.229130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37436 (local address 127.0.97.62:35323)
0927 22:09:28.229340 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:28.229347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:28.231257 (+ 1910us) server_negotiation.cc:407] Connection header received
0927 22:09:28.235390 (+ 4133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:28.235400 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:28.235531 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:28.235610 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:28.237253 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.238458 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.240835 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.241598 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.241708 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.244625 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:28.244634 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:28.244642 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:28.244688 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:28.248650 (+ 3962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.249526 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.249533 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.249537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.249699 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.250181 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.250186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.250190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.250548 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:28.250742 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:28.251387 (+ 645us) server_negotiation.cc:299] Negotiation successful
0927 22:09:28.251640 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:09:28.253950 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.231625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39403 (local address 127.0.0.1:45414)
0927 22:09:28.238179 (+ 6554us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:28.238201 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:28.238470 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:28.238998 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:28.239003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:28.239827 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:28.240154 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.240165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.241665 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.241670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.243717 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.243730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.245015 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.245022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.245195 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.246152 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:28.246174 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:28.247464 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:28.251055 (+ 3591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:28.251060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:28.251063 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:28.251562 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:28.251944 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:28.251949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:28.251952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:28.252113 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:28.252833 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:28.252838 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:28.252990 (+ 152us) client_negotiation.cc:769] Sending connection context
0927 22:09:28.253149 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 22:09:28.253464 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6416,"thread_start_us":6644,"threads_started":1}
I20260927 22:09:28.254082 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.229149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.0.1:37436)
0927 22:09:28.231144 (+ 1995us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:28.231165 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:28.231364 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:28.235844 (+ 4480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:28.235850 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:28.236656 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:28.237065 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.237080 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.238665 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.238673 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.240666 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.240682 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.242009 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.242017 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.242284 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.243050 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:28.243077 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:28.244396 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:28.248820 (+ 4424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:28.248827 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:28.248833 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:28.249386 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:28.249849 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:28.249854 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:28.249859 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:28.250033 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:28.250723 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:28.250730 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:28.250972 (+ 242us) client_negotiation.cc:769] Sending connection context
0927 22:09:28.253299 (+ 2327us) client_negotiation.cc:240] Negotiation successful
0927 22:09:28.253640 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1797,"mutex_wait_us":4,"thread_start_us":1624,"threads_started":1}
I20260927 22:09:28.254267 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.231337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45414 (local address 127.0.97.61:39403)
0927 22:09:28.231498 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:28.231504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:28.238270 (+ 6766us) server_negotiation.cc:407] Connection header received
0927 22:09:28.238630 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:28.238635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:28.238697 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:28.238777 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:28.240311 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.241414 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.243859 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.244597 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.244713 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.247662 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:28.247668 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:28.247673 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:28.247710 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:28.250905 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.251675 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.251679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.251683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.251823 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.252237 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.252242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.252245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.252586 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:28.252770 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:28.253279 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 22:09:28.253770 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":302,"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 27 22:09:28 dist-test-slave-mp03 krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546965, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 22:09:28.281307 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.261545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45422 (local address 127.0.97.61:39403)
0927 22:09:28.261750 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:28.261760 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:28.262434 (+ 674us) server_negotiation.cc:407] Connection header received
0927 22:09:28.262838 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:28.262845 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:28.262953 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:28.263056 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:28.264586 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.265764 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.267928 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.268664 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.268794 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.273431 (+ 4637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:28.273442 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:28.273450 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:28.273505 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:28.277148 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.278084 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.278090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.278093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.278266 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.278757 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.278761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.278763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.279108 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:28.279300 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:28.280582 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0927 22:09:28.280907 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:09:28.281657 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.261373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39403 (local address 127.0.0.1:45422)
0927 22:09:28.262300 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:28.262350 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:28.262577 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:28.263131 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:28.263141 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:28.264056 (+ 915us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:28.264422 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.264438 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.265920 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.265928 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.267773 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.267791 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.268848 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.268856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.269097 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.269987 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:28.270016 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:28.273184 (+ 3168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:28.277311 (+ 4127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:28.277325 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:28.277334 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:28.277936 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:28.278395 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:28.278400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:28.278411 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:28.278635 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:28.279291 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:28.279300 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:28.279517 (+ 217us) client_negotiation.cc:769] Sending connection context
0927 22:09:28.279766 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 22:09:28.280056 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":489,"threads_started":1}
I20260927 22:09:28.283921 6536 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:39403: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 39403
}
attrs {
promote: true
}
}
dest_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d"
cas_config_opid_index: -1
I20260927 22:09:28.285065 6536 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260927 22:09:28.287658 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40003
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35323,127.0.97.61:39403,127.0.97.60:40003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:09:28.296389 7041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:28.300261 7041 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [term 1 FOLLOWER]: Refusing update from remote peer ae6791bd85ae4fceb7d3ac0a20c86d2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 22:09:28.302376 7096 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ef79440c0b4c4fba8b6433b957ca03c3 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: true } } }
I20260927 22:09:28.305974 7097 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:09:28.309087 7096 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:39403
I20260927 22:09:28.309794 6999 master.cc:537] Master@127.0.97.61:39403 shutting down...
I20260927 22:09:28.310025 7098 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: true } } }
I20260927 22:09:28.310338 7098 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:28.338209 7041 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ef79440c0b4c4fba8b6433b957ca03c3 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: true } } }
I20260927 22:09:28.355949 7104 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: true } } }
I20260927 22:09:28.356184 7104 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [sys.catalog]: This master's current role is: LEARNER
I20260927 22:09:28.355254 7108 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: true } } }
I20260927 22:09:28.356384 7108 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [sys.catalog]: This master's current role is: LEARNER
I20260927 22:09:28.355703 7073 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae6791bd85ae4fceb7d3ac0a20c86d2d. Latest consensus state: current_term: 1 leader_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: true } } }
I20260927 22:09:28.356559 7073 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [sys.catalog]: This master's current role is: LEARNER
I20260927 22:09:28.358045 6999 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [term 1 LEARNER]: Raft consensus shutting down.
I20260927 22:09:28.358520 6999 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [term 1 LEARNER]: Raft consensus is shut down!
I20260927 22:09:28.358714 6999 tablet_replica.cc:330] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3: stopping tablet replica
I20260927 22:09:28.376844 6999 master.cc:552] Master@127.0.97.61:39403 shutdown complete.
I20260927 22:09:28.376953 6999 master_runner.cc:305] Clearing existing system tablet
I20260927 22:09:28.377417 6999 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:28.377528 6999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:28.380635 6999 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 22:09:28.382403 7110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:28.382804 6999 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:28.383067 6999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 22:09:28.383178 6999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/wal
uuid: "ef79440c0b4c4fba8b6433b957ca03c3"
format_stamp: "Formatted at 2026-09-27 22:09:28 on dist-test-slave-mp03"
server_key: "e350ff6d53bdd74a6daacc5b6cc5fc1c"
server_key_iv: "bfae5644e7a994eaf51fe64c7bff2d5b"
server_key_version: "encryptionkey@0"
I20260927 22:09:28.387657 6999 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:28.394810 6999 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 22:09:28.394996 6999 log.cc:1196] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 22:09:28.395486 6999 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3: Deleting consensus metadata
I20260927 22:09:28.395763 6999 master_runner.cc:315] Copying system tablet from 127.0.97.62:35323
I20260927 22:09:28.398512 6999 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:35323
I20260927 22:09:28.416210 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.399426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55552 (local address 127.0.97.62:35323)
0927 22:09:28.399618 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:28.399625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:28.400157 (+ 532us) server_negotiation.cc:407] Connection header received
0927 22:09:28.400343 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:28.400351 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:28.400468 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:28.400603 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:28.401683 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.403032 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.405083 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.405877 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.406041 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.408951 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:28.408959 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:28.408964 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:28.409006 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:28.412682 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.413591 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.413596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.413600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.413756 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.414222 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.414227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.414230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.414650 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:28.414839 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:28.415379 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 22:09:28.415655 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:09:28.416353 7116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.399408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.0.1:55552)
0927 22:09:28.400014 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:28.400035 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:28.400191 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:28.400640 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:28.400645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:28.401161 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:28.401499 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.401522 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.403238 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.403244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.404891 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.404912 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.406318 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.406326 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.406562 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.407386 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:28.407416 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:28.408706 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:28.412842 (+ 4136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:28.412849 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:28.412855 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:28.413445 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:28.413885 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:28.413891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:28.413895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:28.414093 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:28.414783 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:28.414790 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:28.415048 (+ 258us) client_negotiation.cc:769] Sending connection context
0927 22:09:28.415272 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 22:09:28.415742 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":342,"threads_started":1}
I20260927 22:09:28.417471 6557 tablet_copy_service.cc:140] P ae6791bd85ae4fceb7d3ac0a20c86d2d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ef79440c0b4c4fba8b6433b957ca03c3 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:55552)
I20260927 22:09:28.417763 6557 tablet_copy_service.cc:161] P ae6791bd85ae4fceb7d3ac0a20c86d2d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ef79440c0b4c4fba8b6433b957ca03c3 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:55552: session id = ef79440c0b4c4fba8b6433b957ca03c3-00000000000000000000000000000000
I20260927 22:09:28.423066 6557 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 22:09:28.425582 6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 22:09:28.432715 6999 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3: tablet copy: Starting download of 0 data blocks...
I20260927 22:09:28.433094 6999 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3: tablet copy: Starting download of 1 WAL segments...
I20260927 22:09:28.436347 6999 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 22:09:28.441663 6557 tablet_copy_service.cc:342] P ae6791bd85ae4fceb7d3ac0a20c86d2d: Request end of tablet copy session ef79440c0b4c4fba8b6433b957ca03c3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:55552
I20260927 22:09:28.441885 6557 tablet_copy_service.cc:433] P ae6791bd85ae4fceb7d3ac0a20c86d2d: ending tablet copy session ef79440c0b4c4fba8b6433b957ca03c3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ef79440c0b4c4fba8b6433b957ca03c3
I20260927 22:09:28.454612 6999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:28.457440 7117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:28.470099 7118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:28.471853 7120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:28.472702 6999 server_base.cc:1034] running on GCE node
I20260927 22:09:28.473022 6999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:28.473691 6999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:28.474943 6999 hybrid_clock.cc:648] HybridClock initialized: now 1790546968474891 us; error 34 us; skew 500 ppm
I20260927 22:09:28.475172 6999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:28 dist-test-slave-mp03 krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:28.477160 6999 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260927 22:09:28.478489 6999 webserver.cc:460] Webserver started at http://127.0.97.61:43387/ using document root <none> and password file <none>
I20260927 22:09:28.478837 6999 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:28.478928 6999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:28.481335 6999 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 22:09:28.482949 7127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:28.483372 6999 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260927 22:09:28.483507 6999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/wal
uuid: "ef79440c0b4c4fba8b6433b957ca03c3"
format_stamp: "Formatted at 2026-09-27 22:09:28 on dist-test-slave-mp03"
server_key: "e350ff6d53bdd74a6daacc5b6cc5fc1c"
server_key_iv: "bfae5644e7a994eaf51fe64c7bff2d5b"
server_key_version: "encryptionkey@0"
I20260927 22:09:28.483677 6999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:28.490912 6999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:28.495066 6999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:28.520589 6999 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:39403
I20260927 22:09:28.520620 7178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:39403 every 8 connection(s)
I20260927 22:09:28.521502 6999 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-1/data/info.pb
I20260927 22:09:28.529443 7179 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:09:28.535063 7179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3: Bootstrap starting.
I20260927 22:09:28.546772 7179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:09:28.547214 7179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3: Bootstrap complete.
I20260927 22:09:28.548069 7179 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: true } }
I20260927 22:09:28.548346 7179 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef79440c0b4c4fba8b6433b957ca03c3, State: Initialized, Role: LEARNER
I20260927 22:09:28.548626 7179 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: true } }
I20260927 22:09:28.549403 7182 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: true } } }
I20260927 22:09:28.549749 7179 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:09:28.549820 7182 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [sys.catalog]: This master's current role is: LEARNER
I20260927 22:09:28.559001 7193 catalog_manager.cc:1259] Loaded cluster ID: 0a21bad7669c4c3ea45c68ac34cabb13
I20260927 22:09:28.559124 7193 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3: loading cluster ID for follower catalog manager: success
I20260927 22:09:28.561509 7193 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3: acquiring CA information for follower catalog manager: success
I20260927 22:09:28.563815 7193 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:09:28.652981 6993 heartbeater.cc:503] Master 127.0.97.62:35323 was elected leader, sending a full tablet report...
I20260927 22:09:28.735363 7096 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: true } } }
I20260927 22:09:28.735656 7096 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: This master's current role is: LEADER
W20260927 22:09:28.741168 7099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:28.741537 7099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:28.741688 7099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 22:09:28.742905 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.735758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39403 (local address 127.0.0.1:48224)
0927 22:09:28.736059 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:28.736083 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:28.736192 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:28.736986 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:28.736991 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:28.737026 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:09:28.737226 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.737237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.738570 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.738577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.740674 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.740689 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.742053 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.742060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.742257 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.742339 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 22:09:28.742436 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 22:09:28.742611 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20260927 22:09:28.742980 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.736072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48224 (local address 127.0.97.61:39403)
0927 22:09:28.736650 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:28.736654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:28.736671 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:09:28.736729 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:28.736732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:28.736797 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:28.736896 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:09:28.737398 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.738403 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.740833 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.741865 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.741974 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.742227 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:28.742455 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 22:09:28.742622 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":308,"threads_started":1}
I20260927 22:09:28.744086 7182 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae6791bd85ae4fceb7d3ac0a20c86d2d. Latest consensus state: current_term: 1 leader_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: true } } }
I20260927 22:09:28.744365 7182 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [sys.catalog]: This master's current role is: LEARNER
I20260927 22:09:28.744983 7098 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d: attempting to promote NON_VOTER ef79440c0b4c4fba8b6433b957ca03c3 to VOTER
I20260927 22:09:28.745814 7098 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } }
I20260927 22:09:28.747969 7154 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [term 1 LEARNER]: Refusing update from remote peer ae6791bd85ae4fceb7d3ac0a20c86d2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 22:09:28.748481 7096 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 22:09:28.751610 7098 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ef79440c0b4c4fba8b6433b957ca03c3 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } }
W20260927 22:09:28.752233 7099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:09:28.752380 7099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:28.752483 7099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:28.752568 7099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:09:28.752631 7099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:09:28.752480 7154 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ef79440c0b4c4fba8b6433b957ca03c3 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } }
I20260927 22:09:28.755615 7098 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } }
I20260927 22:09:28.755867 7098 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:28.755891 7096 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } }
I20260927 22:09:28.756281 7096 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:28.756697 7182 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } }
I20260927 22:09:28.756924 7182 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:09:28.764093 7099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35323,127.0.97.61:39403,127.0.97.60:40003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:28.765558 7099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:28.766976 7099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:28.779408 7203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:28.779524 7200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:28.779551 7201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:28.779985 7099 server_base.cc:1034] running on GCE node
I20260927 22:09:28.780983 7099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:28.782433 7099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:28.783674 7099 hybrid_clock.cc:648] HybridClock initialized: now 1790546968783651 us; error 44 us; skew 500 ppm
I20260927 22:09:28.784065 7099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:28 dist-test-slave-mp03 krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:28.787285 7099 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260927 22:09:28.788717 7099 webserver.cc:460] Webserver started at http://127.0.97.60:42741/ using document root <none> and password file <none>
I20260927 22:09:28.789417 7099 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:28.789569 7099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:28.789871 7099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:28.792872 7099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/data/instance:
uuid: "4cee22109f194af68fb1eb895cce38c7"
format_stamp: "Formatted at 2026-09-27 22:09:28 on dist-test-slave-mp03"
server_key: "b8dd574b28d620ee9b093dc841e4a692"
server_key_iv: "7d9eb377f5d9765aef98fabae4bb47b9"
server_key_version: "encryptionkey@0"
I20260927 22:09:28.793645 7099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/wal/instance:
uuid: "4cee22109f194af68fb1eb895cce38c7"
format_stamp: "Formatted at 2026-09-27 22:09:28 on dist-test-slave-mp03"
server_key: "b8dd574b28d620ee9b093dc841e4a692"
server_key_iv: "7d9eb377f5d9765aef98fabae4bb47b9"
server_key_version: "encryptionkey@0"
I20260927 22:09:28.799304 7099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 22:09:28.802816 7210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:28.804193 7099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:09:28.804380 7099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/wal
uuid: "4cee22109f194af68fb1eb895cce38c7"
format_stamp: "Formatted at 2026-09-27 22:09:28 on dist-test-slave-mp03"
server_key: "b8dd574b28d620ee9b093dc841e4a692"
server_key_iv: "7d9eb377f5d9765aef98fabae4bb47b9"
server_key_version: "encryptionkey@0"
I20260927 22:09:28.804615 7099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:28.824999 7099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:28.829301 7099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:28.829598 7099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:28.849895 7099 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:40003
I20260927 22:09:28.849939 7261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:40003 every 8 connection(s)
I20260927 22:09:28.851533 7099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/data/info.pb
I20260927 22:09:28.855708 7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:28.856864 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 7099
I20260927 22:09:28.866127 7262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:09:28 dist-test-slave-mp03 krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 27 22:09:28 dist-test-slave-mp03 krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:09:28.882658 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.859273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42090 (local address 127.0.97.60:40003)
0927 22:09:28.860593 (+ 1320us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:28.860611 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:28.860989 (+ 378us) server_negotiation.cc:407] Connection header received
0927 22:09:28.862869 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:28.862893 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:28.863345 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:28.863707 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:28.865515 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.866892 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.868643 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.869759 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.869933 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.874187 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:28.874194 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:28.874205 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:28.874240 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:28.877771 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.878649 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.878658 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.878667 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.878859 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.879243 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.879248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.879252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.879984 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:28.880290 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:28.880782 (+ 492us) server_negotiation.cc:299] Negotiation successful
0927 22:09:28.881230 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":421,"threads_started":1}
I20260927 22:09:28.884811 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.869111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.0.1:55554)
0927 22:09:28.869932 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:28.869979 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:28.870212 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:28.870642 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:28.870649 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:28.871229 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:28.871448 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.871456 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.872870 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.872874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.873968 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.873981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.874794 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.874800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.874928 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.875529 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:28.875552 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:28.878377 (+ 2825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:28.882100 (+ 3723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:28.882109 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:28.882114 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:28.882521 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:28.882988 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:28.882993 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:28.882996 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:28.883136 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:28.883791 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:28.883798 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:28.883983 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 22:09:28.884212 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 22:09:28.884467 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":438,"threads_started":1}
I20260927 22:09:28.884857 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.869155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55554 (local address 127.0.97.62:35323)
0927 22:09:28.869320 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:28.869324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:28.870071 (+ 747us) server_negotiation.cc:407] Connection header received
0927 22:09:28.870371 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:28.870376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:28.870440 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:28.870559 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:28.871617 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.872290 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.874125 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.874537 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.874719 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.878624 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:28.878631 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:28.878636 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:28.878680 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:28.881971 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.882680 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.882687 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.882691 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.882835 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.883279 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.883285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.883289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.883652 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:28.883756 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:28.884327 (+ 571us) server_negotiation.cc:299] Negotiation successful
0927 22:09:28.884560 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:09:28.886143 7262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:09:28 dist-test-slave-mp03 krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 22:09:28.899187 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.886775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39403 (local address 127.0.0.1:48236)
0927 22:09:28.886928 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:28.886945 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:28.887059 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:28.887390 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:28.887394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:28.887744 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:28.887875 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.887882 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.889315 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.889321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.890253 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.890264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.891073 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.891078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.891247 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.891799 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:28.891815 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:28.893798 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:28.896765 (+ 2967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:28.896770 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:28.896774 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:28.897174 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:28.897560 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:28.897566 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:28.897570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:28.897710 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:28.898220 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:28.898225 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:28.898375 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 22:09:28.898515 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 22:09:28.898837 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 22:09:28.899207 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.886935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48236 (local address 127.0.97.61:39403)
0927 22:09:28.887115 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:28.887123 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:28.887142 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:09:28.887195 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:28.887200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:28.887267 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:28.887371 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:28.888019 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.889171 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.890402 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.890904 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.891051 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.893996 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:28.894007 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:28.894012 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:28.894046 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:28.896610 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.897300 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.897306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.897310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.897449 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.897814 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.897821 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.897825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.898107 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:28.898259 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:28.898621 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 22:09:28.898899 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:09:28.900264 7262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40003 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:09:28 dist-test-slave-mp03 krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260927 22:09:28.912838 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.901034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40003 (local address 127.0.0.1:42102)
0927 22:09:28.901273 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:28.901286 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:28.901374 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:28.901614 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:28.901617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:28.901924 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:28.902060 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.902067 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.903053 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.903058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.904058 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.904069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.904739 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.904744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.904883 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.905591 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:28.905610 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:28.907842 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:28.910767 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:28.910771 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:28.910773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:28.911158 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:28.911499 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:28.911503 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:28.911505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:28.911651 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:28.912084 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:28.912087 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:28.912207 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 22:09:28.912321 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 22:09:28.912529 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260927 22:09:28.912874 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.901034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42102 (local address 127.0.97.60:40003)
0927 22:09:28.901180 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:28.901184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:28.901353 (+ 169us) server_negotiation.cc:407] Connection header received
0927 22:09:28.901458 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:28.901461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:28.901515 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:28.901583 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:28.902169 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.902929 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.904181 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.904570 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.904666 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.908021 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:28.908026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:28.908029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:28.908056 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:28.910640 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.911256 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.911260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.911262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.911377 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.911738 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.911741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.911743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.911988 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:28.912093 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:28.912443 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 22:09:28.912625 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":51}
I20260927 22:09:28.918044 7262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7: Bootstrap starting.
I20260927 22:09:28.921437 7262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:28.922576 7262 log.cc:824] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:28.925268 7262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7: No bootstrap required, opened a new log
I20260927 22:09:28.933332 7262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40003 } }
I20260927 22:09:28.933661 7262 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:28.933799 7262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cee22109f194af68fb1eb895cce38c7, State: Initialized, Role: FOLLOWER
I20260927 22:09:28.934396 7262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40003 } }
I20260927 22:09:28.935271 7268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40003 } } }
I20260927 22:09:28.935655 7268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:09:28.936362 7262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 22:09:28.946702 7279 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:09:28.946851 7279 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:09:28.953720 7099 master_runner.cc:428] Detected that this master 4cee22109f194af68fb1eb895cce38c7 is joining an existing cluster
I20260927 22:09:28.953887 7099 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:40003
I20260927 22:09:28.980930 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.957937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55560 (local address 127.0.97.62:35323)
0927 22:09:28.958141 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:28.958148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:28.963260 (+ 5112us) server_negotiation.cc:407] Connection header received
0927 22:09:28.963797 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:28.963803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:28.963887 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:28.963988 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:28.967020 (+ 3032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.968232 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.970373 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.971059 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.971157 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.974008 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:28.974015 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:28.974021 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:28.974059 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:28.977546 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.978419 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.978425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.978430 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.978585 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.979121 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.979129 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.979134 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.979474 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:28.979622 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:28.980288 (+ 666us) server_negotiation.cc:299] Negotiation successful
0927 22:09:28.980529 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:09:28.981655 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.963581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48246 (local address 127.0.97.61:39403)
0927 22:09:28.963759 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:28.963765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:28.965813 (+ 2048us) server_negotiation.cc:407] Connection header received
0927 22:09:28.965949 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:28.965957 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:28.966041 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:28.966139 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:28.967577 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.968643 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.970662 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.971683 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.971781 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.975010 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:28.975016 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:28.975020 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:28.975062 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:28.978572 (+ 3510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.979398 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.979404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.979407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.979558 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.980006 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.980010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.980013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.980332 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:28.980472 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:28.981081 (+ 609us) server_negotiation.cc:299] Negotiation successful
0927 22:09:28.981301 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:09:28.982438 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.958429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42104 (local address 127.0.97.60:40003)
0927 22:09:28.958640 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:28.958646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:28.964007 (+ 5361us) server_negotiation.cc:407] Connection header received
0927 22:09:28.964258 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:28.964266 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:28.964350 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:28.964426 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:28.966479 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.967719 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.969659 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.970387 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.970480 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.974264 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:28.974270 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:28.974275 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:28.974309 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:28.978824 (+ 4515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.979636 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.979640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.979643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.979760 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:28.980168 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:28.980172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:28.980176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:28.980544 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:28.980691 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:28.981528 (+ 837us) server_negotiation.cc:299] Negotiation successful
0927 22:09:28.981975 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":61,"spinlock_wait_cycles":7936}
I20260927 22:09:28.982036 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.958271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.0.1:55560)
0927 22:09:28.963121 (+ 4850us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:28.963149 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:28.963611 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:28.964235 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:28.964239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:28.966235 (+ 1996us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:28.966837 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.966849 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.968447 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.968452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.970177 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.970196 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.971436 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.971442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.971629 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.972274 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:28.972304 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:28.973806 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:28.977705 (+ 3899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:28.977712 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:28.977720 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:28.978281 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:28.978743 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:28.978747 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:28.978750 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:28.978981 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:28.979703 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:28.979707 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:28.979935 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 22:09:28.981240 (+ 1305us) client_negotiation.cc:240] Negotiation successful
0927 22:09:28.981566 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4693,"mutex_wait_us":75,"spinlock_wait_cycles":32896,"thread_start_us":4522,"threads_started":1}
I20260927 22:09:28.983134 7287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.963890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39403 (local address 127.0.0.1:48246)
0927 22:09:28.965695 (+ 1805us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:28.965716 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:28.965834 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:28.966329 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:28.966334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:28.967079 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:28.967397 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.967407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.968836 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.968844 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.970488 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.970498 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.972058 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.972065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.972259 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.972905 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:28.972926 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:28.974809 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:28.978736 (+ 3927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:28.978741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:28.978744 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:28.979273 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:28.979697 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:28.979701 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:28.979705 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:28.979870 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:28.980549 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:28.980554 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:28.980779 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 22:09:28.981881 (+ 1102us) client_negotiation.cc:240] Negotiation successful
0927 22:09:28.982198 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1677,"mutex_wait_us":5,"thread_start_us":1537,"threads_started":1}
I20260927 22:09:28.984936 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.958316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40003 (local address 127.0.0.1:42104)
0927 22:09:28.963889 (+ 5573us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:28.963906 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:28.964109 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:28.964685 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:28.964692 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:28.965895 (+ 1203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:28.966329 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.966340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.967909 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.967914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.969513 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.969531 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.970801 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.970807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.970981 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.971683 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:28.971763 (+ 80us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:28.974079 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:28.978970 (+ 4891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:28.978975 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:28.978981 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:28.979510 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:28.979898 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:28.979903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:28.979906 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:28.980060 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:28.980770 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:28.980774 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:28.980927 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 22:09:28.982064 (+ 1137us) client_negotiation.cc:240] Negotiation successful
0927 22:09:28.982376 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5436,"mutex_wait_us":31,"spinlock_wait_cycles":45952,"thread_start_us":5101,"threads_started":1}
I20260927 22:09:28.988646 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 6598
Sep 27 22:09:28 dist-test-slave-mp03 krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546965, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 27 22:09:28 dist-test-slave-mp03 krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260927 22:09:29.004696 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.988466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40003 (local address 127.0.0.1:42106)
0927 22:09:28.988727 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:28.988744 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:28.988880 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:28.989298 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:28.989304 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:28.989793 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:28.989994 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.990006 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.991271 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.991277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.992684 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:28.992698 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.993615 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.993621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:28.993782 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.994389 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:28.994406 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:28.997054 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:29.001230 (+ 4176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:29.001236 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:29.001241 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:29.001770 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:29.002721 (+ 951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:29.002727 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:29.002731 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:29.002887 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:29.003581 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:29.003587 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:29.003762 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 22:09:29.003964 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 22:09:29.004263 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260927 22:09:29.005671 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.988531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42106 (local address 127.0.97.60:40003)
0927 22:09:28.988686 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:28.988692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:28.988821 (+ 129us) server_negotiation.cc:407] Connection header received
0927 22:09:28.989036 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:28.989042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:28.989165 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:28.989267 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:28.990151 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.991131 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.992887 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.993459 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.993603 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:28.997264 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:28.997271 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:28.997276 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:28.997312 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:29.001067 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:29.001948 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:29.001957 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:29.001961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:29.002539 (+ 578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:29.003051 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:29.003056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:29.003060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:29.003440 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:29.004950 (+ 1510us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:29.005087 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 22:09:29.005311 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:09:29.007390 6536 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:40003: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4cee22109f194af68fb1eb895cce38c7"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 40003
}
attrs {
promote: true
}
}
dest_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d"
cas_config_opid_index: 6
I20260927 22:09:29.008294 6536 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: true } }
I20260927 22:09:29.010792 7154 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [term 1 FOLLOWER]: Refusing update from remote peer ae6791bd85ae4fceb7d3ac0a20c86d2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 22:09:29.011029 7288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:28.990232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51272 (local address 127.0.97.1:43179)
0927 22:09:28.990963 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:28.990972 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:28.991003 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:09:28.991073 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:28.991082 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:28.991367 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:28.991545 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:28.993144 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.994795 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.996399 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:28.996983 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:28.997124 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:29.002350 (+ 5226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:29.002367 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:29.002388 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:29.002435 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:29.006742 (+ 4307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:29.007605 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:29.007726 (+ 121us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:29.007729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:29.007872 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:29.008317 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:29.008322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:29.008325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:29.008821 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:29.010082 (+ 1261us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:29.010250 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 22:09:29.010515 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":370,"threads_started":1}
I20260927 22:09:29.011412 7096 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 22:09:29.012629 6721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:29.013293 7237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:29.014853 7096 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4cee22109f194af68fb1eb895cce38c7 (127.0.97.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: true } } }
I20260927 22:09:29.017390 7237 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [term 1 FOLLOWER]: Refusing update from remote peer ae6791bd85ae4fceb7d3ac0a20c86d2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 22:09:29.018282 7100 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:09:29.022104 7154 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4cee22109f194af68fb1eb895cce38c7 (127.0.97.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: true } } }
I20260927 22:09:29.025918 7182 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: true } } }
I20260927 22:09:29.026250 7182 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:09:29.027271 7096 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:40003
I20260927 22:09:29.027889 7099 master.cc:537] Master@127.0.97.60:40003 shutting down...
I20260927 22:09:29.027856 7098 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: true } } }
I20260927 22:09:29.028188 7098 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:29.028466 7098 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: true } } }
I20260927 22:09:29.028749 7098 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:29.040642 7237 raft_consensus.cc:858] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40003 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } }
W20260927 22:09:29.040861 7237 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40003 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } }
I20260927 22:09:29.054477 7237 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ef79440c0b4c4fba8b6433b957ca03c3 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: true } } }
I20260927 22:09:29.064695 7268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae6791bd85ae4fceb7d3ac0a20c86d2d. Latest consensus state: current_term: 1 leader_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: true } } }
I20260927 22:09:29.064973 7268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 22:09:29.065263 7292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: true } } }
I20260927 22:09:29.065555 7292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 22:09:29.065549 7096 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: true } } }
I20260927 22:09:29.065884 7096 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:29.065980 7296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: true } } }
I20260927 22:09:29.066335 7296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [sys.catalog]: This master's current role is: NON_PARTICIPANT
W20260927 22:09:29.067157 6523 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 22:09:29.068763 6523 consensus_peers.cc:487] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d -> Peer 4cee22109f194af68fb1eb895cce38c7 (127.0.97.60:40003): Couldn't send request to peer 4cee22109f194af68fb1eb895cce38c7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 22:09:29.069103 7099 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260927 22:09:29.069583 7099 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260927 22:09:29.069765 7099 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7: stopping tablet replica
I20260927 22:09:29.082788 7099 master.cc:552] Master@127.0.97.60:40003 shutdown complete.
I20260927 22:09:29.082948 7099 master_runner.cc:305] Clearing existing system tablet
I20260927 22:09:29.083390 7099 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:29.083617 7099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:29.094841 7099 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.003s sys 0.000s
I20260927 22:09:29.097332 7299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:29.097757 7099 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:29.097998 7099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:09:29.098093 7099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/wal
uuid: "4cee22109f194af68fb1eb895cce38c7"
format_stamp: "Formatted at 2026-09-27 22:09:28 on dist-test-slave-mp03"
server_key: "b8dd574b28d620ee9b093dc841e4a692"
server_key_iv: "7d9eb377f5d9765aef98fabae4bb47b9"
server_key_version: "encryptionkey@0"
I20260927 22:09:29.107002 7099 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:29.114797 7099 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 22:09:29.114998 7099 log.cc:1196] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 22:09:29.115566 7099 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7: Deleting consensus metadata
I20260927 22:09:29.115840 7099 master_runner.cc:315] Copying system tablet from 127.0.97.62:35323
I20260927 22:09:29.118491 7099 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:35323
I20260927 22:09:29.135150 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:29.119380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55576 (local address 127.0.97.62:35323)
0927 22:09:29.119601 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:29.119608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:29.120103 (+ 495us) server_negotiation.cc:407] Connection header received
0927 22:09:29.120388 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:29.120394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:29.120475 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:29.120563 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:29.121983 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:29.123158 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:29.125020 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:29.125663 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:29.125798 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:29.128482 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:29.128489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:29.128494 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:29.128544 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:29.132204 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:29.132991 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:29.132997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:29.133000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:29.133146 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:29.133576 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:29.133581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:29.133584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:29.133941 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:29.134132 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:29.134533 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 22:09:29.134811 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260927 22:09:29.135180 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:29.119299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.0.1:55576)
0927 22:09:29.119988 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:29.120009 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:29.120233 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:29.121050 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:29.121055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:29.121549 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:29.121810 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:29.121821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:29.123335 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:29.123341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:29.124860 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:29.124874 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:29.125838 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:29.125844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:29.126014 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:29.127124 (+ 1110us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:29.127149 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:29.128270 (+ 1121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:29.132354 (+ 4084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:29.132361 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:29.132365 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:29.132862 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:29.133283 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:29.133288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:29.133292 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:29.133457 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:29.134069 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:29.134075 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:29.134263 (+ 188us) client_negotiation.cc:769] Sending connection context
0927 22:09:29.134414 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 22:09:29.134781 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":399,"threads_started":1}
I20260927 22:09:29.136091 6557 tablet_copy_service.cc:140] P ae6791bd85ae4fceb7d3ac0a20c86d2d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4cee22109f194af68fb1eb895cce38c7 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:55576)
I20260927 22:09:29.136255 6557 tablet_copy_service.cc:161] P ae6791bd85ae4fceb7d3ac0a20c86d2d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4cee22109f194af68fb1eb895cce38c7 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:55576: session id = 4cee22109f194af68fb1eb895cce38c7-00000000000000000000000000000000
I20260927 22:09:29.140081 6557 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 22:09:29.141960 7099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 22:09:29.149873 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 6598
I20260927 22:09:29.155212 7099 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7: tablet copy: Starting download of 0 data blocks...
I20260927 22:09:29.155573 7099 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7: tablet copy: Starting download of 1 WAL segments...
I20260927 22:09:29.158053 7099 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 22:09:29.161485 6557 tablet_copy_service.cc:342] P ae6791bd85ae4fceb7d3ac0a20c86d2d: Request end of tablet copy session 4cee22109f194af68fb1eb895cce38c7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:55576
I20260927 22:09:29.161640 6557 tablet_copy_service.cc:433] P ae6791bd85ae4fceb7d3ac0a20c86d2d: ending tablet copy session 4cee22109f194af68fb1eb895cce38c7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4cee22109f194af68fb1eb895cce38c7
I20260927 22:09:29.171670 7099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 22:09:29.173287 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43179
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:35323
--webserver_port=36345
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:35323,127.0.97.61:39403
--tserver_master_addrs=127.0.97.62:35323,127.0.97.61:39403,127.0.97.60:40003 with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:29.180938 7306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:29.182207 7307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:29.190438 7310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:29.191428 7099 server_base.cc:1034] running on GCE node
I20260927 22:09:29.191779 7099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:29.192502 7099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:29.193724 7099 hybrid_clock.cc:648] HybridClock initialized: now 1790546969193694 us; error 59 us; skew 500 ppm
I20260927 22:09:29.193961 7099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:29 dist-test-slave-mp03 krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546969, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:29.195999 7099 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260927 22:09:29.199491 7099 webserver.cc:460] Webserver started at http://127.0.97.60:33061/ using document root <none> and password file <none>
I20260927 22:09:29.199788 7099 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:29.199900 7099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:29.202251 7099 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 22:09:29.203934 7317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:29.204368 7099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 22:09:29.204506 7099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/wal
uuid: "4cee22109f194af68fb1eb895cce38c7"
format_stamp: "Formatted at 2026-09-27 22:09:28 on dist-test-slave-mp03"
server_key: "b8dd574b28d620ee9b093dc841e4a692"
server_key_iv: "7d9eb377f5d9765aef98fabae4bb47b9"
server_key_version: "encryptionkey@0"
I20260927 22:09:29.204696 7099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:29.233194 7099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:29.236750 7099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:29.272101 7099 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:40003
I20260927 22:09:29.272130 7368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:40003 every 8 connection(s)
I20260927 22:09:29.273042 7099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/master-2/data/info.pb
I20260927 22:09:29.280210 7369 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:09:29.285109 7369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7: Bootstrap starting.
I20260927 22:09:29.297340 7369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:09:29.297957 7369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7: Bootstrap complete.
I20260927 22:09:29.299190 7369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: true } }
I20260927 22:09:29.299559 7369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4cee22109f194af68fb1eb895cce38c7, State: Initialized, Role: LEARNER
I20260927 22:09:29.299860 7369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: true } }
I20260927 22:09:29.301987 7372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: true } } }
I20260927 22:09:29.302405 7369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:09:29.302416 7372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [sys.catalog]: This master's current role is: LEARNER
I20260927 22:09:29.318781 7383 catalog_manager.cc:1259] Loaded cluster ID: 0a21bad7669c4c3ea45c68ac34cabb13
I20260927 22:09:29.318902 7383 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7: loading cluster ID for follower catalog manager: success
I20260927 22:09:29.322392 7383 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7: acquiring CA information for follower catalog manager: success
I20260927 22:09:29.324827 7383 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:09:29.513371 7098 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: true } } }
I20260927 22:09:29.513731 7098 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:29.521207 7384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:29.513726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40003 (local address 127.0.0.1:42114)
0927 22:09:29.514738 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:29.514763 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:29.514927 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:29.515385 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:29.515390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:29.515426 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:09:29.515682 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:29.515696 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:29.516936 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:29.516944 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:29.519090 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:29.519107 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:29.520374 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:29.520380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:29.520571 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:29.520606 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:09:29.520719 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 22:09:29.520854 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"thread_start_us":738,"threads_started":1}
I20260927 22:09:29.521193 7385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:29.513848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42114 (local address 127.0.97.60:40003)
0927 22:09:29.515068 (+ 1220us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:29.515073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:29.515090 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:09:29.515149 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:29.515152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:29.515223 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:29.515326 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:09:29.515845 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:29.516773 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:29.519249 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:29.520120 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:29.520289 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:29.520534 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:29.520741 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 22:09:29.520862 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1111,"thread_start_us":984,"threads_started":1}
I20260927 22:09:29.522224 7344 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260927 22:09:29.522675 7372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae6791bd85ae4fceb7d3ac0a20c86d2d. Latest consensus state: current_term: 1 leader_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: true } } }
I20260927 22:09:29.522927 7372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [sys.catalog]: This master's current role is: LEARNER
I20260927 22:09:29.523316 7098 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d: attempting to promote NON_VOTER 4cee22109f194af68fb1eb895cce38c7 to VOTER
I20260927 22:09:29.523924 7098 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: false } }
I20260927 22:09:29.526029 7344 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [term 1 LEARNER]: Refusing update from remote peer ae6791bd85ae4fceb7d3ac0a20c86d2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 22:09:29.526031 7154 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [term 1 FOLLOWER]: Refusing update from remote peer ae6791bd85ae4fceb7d3ac0a20c86d2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 22:09:29.526562 7100 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 22:09:29.526948 7096 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 22:09:29.530187 7096 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4cee22109f194af68fb1eb895cce38c7 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: false } } }
I20260927 22:09:29.531975 7343 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4cee22109f194af68fb1eb895cce38c7 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: false } } }
I20260927 22:09:29.534663 7154 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4cee22109f194af68fb1eb895cce38c7 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: false } } }
I20260927 22:09:29.536285 7372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: false } } }
I20260927 22:09:29.536566 7372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:09:29.537556 7096 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: false } } }
I20260927 22:09:29.537889 7096 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:29.538266 7100 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: false } } }
I20260927 22:09:29.538511 7100 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:29.539026 7098 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: false } } }
I20260927 22:09:29.539398 7098 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:29.540004 7387 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: false } } }
I20260927 22:09:29.540388 7387 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 22:09:29.569347 7309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:29.569622 7309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:29.569803 7309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:29.576907 7309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:29.577020 7309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:29.577226 7309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:09:29.587461 7309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36345
--tserver_master_addrs=127.0.97.62:35323,127.0.97.61:39403,127.0.97.60:40003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:29.588833 7309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:29.590569 7309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:29.603855 7392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:29.603967 7395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:29.603826 7393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:29.605592 7309 server_base.cc:1034] running on GCE node
I20260927 22:09:29.606398 7309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:29.607528 7309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:29.608760 7309 hybrid_clock.cc:648] HybridClock initialized: now 1790546969608699 us; error 56 us; skew 500 ppm
I20260927 22:09:29.609050 7309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:29 dist-test-slave-mp03 krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546969, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:29.612318 7309 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 22:09:29.613643 7309 webserver.cc:460] Webserver started at http://127.0.97.1:36345/ using document root <none> and password file <none>
I20260927 22:09:29.614286 7309 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:29.614391 7309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:29.619577 7309 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:09:29.622188 7402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:29.623574 7309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:09:29.623860 7309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "100d5e2a45074ac29b114b3bf0e3f693"
format_stamp: "Formatted at 2026-09-27 22:09:26 on dist-test-slave-mp03"
server_key: "35346e211cafd6ae54bf5650c0773981"
server_key_iv: "c7265530678a053eda5f55157e49fe4d"
server_key_version: "encryptionkey@0"
I20260927 22:09:29.624819 7309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:29.659852 7309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:29.664307 7309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:29.664587 7309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:29.665535 7309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:29.666924 7309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:29.667047 7309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:29.667182 7309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:29.667269 7309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:29.702530 7309 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43179
I20260927 22:09:29.702692 7514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43179 every 8 connection(s)
I20260927 22:09:29.704136 7309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:09:29.714100 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 7309
I20260927 22:09:29.714459 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 6732
Sep 27 22:09:29 dist-test-slave-mp03 krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546969, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 27 22:09:29 dist-test-slave-mp03 krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546969, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 27 22:09:29 dist-test-slave-mp03 krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546969, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:09:29.736562 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:29.708240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40003 (local address 127.0.97.1:41191)
0927 22:09:29.709177 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:29.709230 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:29.711287 (+ 2057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:29.711952 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:29.711960 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:29.713042 (+ 1082us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:29.714641 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:29.714658 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:29.718091 (+ 3433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:29.718099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:29.720303 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:29.720323 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:29.721894 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:29.721902 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:29.722158 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:29.722944 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:29.722980 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:29.728101 (+ 5121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:29.733274 (+ 5173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:29.733278 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:29.733282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:29.733811 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:29.734277 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:29.734285 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:29.734289 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:29.734502 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:29.735141 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:29.735146 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:29.735294 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 22:09:29.735456 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 22:09:29.735729 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":228,"spinlock_wait_cycles":1920,"thread_start_us":414,"threads_started":1}
I20260927 22:09:29.736928 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:29.709870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49871 (local address 127.0.97.61:39403)
0927 22:09:29.710900 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:29.710904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:29.710920 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:09:29.711618 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:29.711624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:29.711697 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:29.712259 (+ 562us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:29.714835 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:29.716187 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:29.719924 (+ 3737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:29.720823 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:29.727485 (+ 6662us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:29.727740 (+ 255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:29.727745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:29.727748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:29.727772 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:29.731493 (+ 3721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:29.732378 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:29.732384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:29.732387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:29.732512 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:29.732950 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:29.732956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:29.732961 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:29.733318 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:29.733457 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:29.736326 (+ 2869us) server_negotiation.cc:299] Negotiation successful
0927 22:09:29.736564 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":921,"thread_start_us":783,"threads_started":1}
I20260927 22:09:29.737476 7385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:29.707641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41191 (local address 127.0.97.60:40003)
0927 22:09:29.707850 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:29.707857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:29.709340 (+ 1483us) server_negotiation.cc:407] Connection header received
0927 22:09:29.711524 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:29.711530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:29.711626 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:29.711739 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:29.716449 (+ 4710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:29.717862 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:29.720527 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:29.721396 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:29.721532 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:29.728483 (+ 6951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:29.728492 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:29.728498 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:29.728543 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:29.733143 (+ 4600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:29.733969 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:29.733974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:29.733978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:29.734138 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:29.734666 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:29.734673 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:29.734679 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:29.735018 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:29.735163 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:29.736900 (+ 1737us) server_negotiation.cc:299] Negotiation successful
0927 22:09:29.737126 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 22:09:29.736586 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:29.709735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39403 (local address 127.0.97.1:49871)
0927 22:09:29.710315 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:29.710337 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:29.711287 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:29.711975 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:29.711980 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:29.713046 (+ 1066us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:29.714575 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:29.714585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:29.716426 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:29.716435 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:29.719685 (+ 3250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:29.719704 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:29.721077 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:29.721084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:29.721362 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:29.722181 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:29.722219 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:29.727342 (+ 5123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:29.731628 (+ 4286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:29.731635 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:29.731649 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:29.732251 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:29.732623 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:29.732629 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:29.732632 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:29.732806 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:29.733525 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:29.733533 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:29.733969 (+ 436us) client_negotiation.cc:769] Sending connection context
0927 22:09:29.734344 (+ 375us) client_negotiation.cc:240] Negotiation successful
0927 22:09:29.734971 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":185,"spinlock_wait_cycles":49152,"thread_start_us":245,"threads_started":1}
I20260927 22:09:29.739354 7517 heartbeater.cc:347] Connected to a master server at 127.0.97.61:39403
I20260927 22:09:29.739385 7515 heartbeater.cc:347] Connected to a master server at 127.0.97.60:40003
I20260927 22:09:29.739678 7517 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:29.739684 7515 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:29.740589 7517 heartbeater.cc:511] Master 127.0.97.61:39403 requested a full tablet report, sending...
I20260927 22:09:29.741108 7515 heartbeater.cc:511] Master 127.0.97.60:40003 requested a full tablet report, sending...
I20260927 22:09:29.741317 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:29.707571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55205 (local address 127.0.97.62:35323)
0927 22:09:29.708202 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:29.708208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:29.709544 (+ 1336us) server_negotiation.cc:407] Connection header received
0927 22:09:29.711618 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:29.711624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:29.711697 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:29.711801 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:29.714754 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:29.716111 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:29.724954 (+ 8843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:29.725974 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:29.726138 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:29.731084 (+ 4946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:29.731092 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:29.731096 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:29.731142 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:29.734770 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:29.737272 (+ 2502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:29.737278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:29.737281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:29.737420 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:29.739302 (+ 1882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:29.739308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:29.739311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:29.739645 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:29.739839 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:29.740423 (+ 584us) server_negotiation.cc:299] Negotiation successful
0927 22:09:29.740944 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":387,"threads_started":1}
I20260927 22:09:29.743016 7144 ts_manager.cc:194] Registered new tserver with Master: 100d5e2a45074ac29b114b3bf0e3f693 (127.0.97.1:43179)
I20260927 22:09:29.743753 7334 ts_manager.cc:194] Registered new tserver with Master: 100d5e2a45074ac29b114b3bf0e3f693 (127.0.97.1:43179)
I20260927 22:09:29.743940 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:29.708240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.97.1:55205)
0927 22:09:29.709177 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:29.709228 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:29.711267 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:29.712160 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:29.712171 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:29.713056 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:29.714552 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:29.714579 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:29.720923 (+ 6344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:29.720931 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:29.723787 (+ 2856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:29.723800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:29.726190 (+ 2390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:29.726206 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:29.726423 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:29.727105 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:29.727132 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:29.730405 (+ 3273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:29.735211 (+ 4806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:29.735215 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:29.735219 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:29.735654 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:29.738965 (+ 3311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:29.738970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:29.738974 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:29.739162 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:29.739939 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:29.739944 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:29.740070 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 22:09:29.740240 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 22:09:29.740544 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":128,"spinlock_wait_cycles":49280,"thread_start_us":383,"threads_started":1}
I20260927 22:09:29.744910 7516 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35323
I20260927 22:09:29.745035 7516 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:29.745366 7516 heartbeater.cc:511] Master 127.0.97.62:35323 requested a full tablet report, sending...
Sep 27 22:09:29 dist-test-slave-mp03 krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260927 22:09:29.746668 6536 ts_manager.cc:194] Re-registered known tserver with Master: 100d5e2a45074ac29b114b3bf0e3f693 (127.0.97.1:43179)
I20260927 22:09:29.747438 6536 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55205
I20260927 22:09:29.753576 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:29.715528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47934 (local address 127.0.97.2:38917)
0927 22:09:29.726743 (+ 11215us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:29.726754 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:29.733317 (+ 6563us) server_negotiation.cc:407] Connection header received
0927 22:09:29.733402 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:29.733411 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:29.733704 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:29.733879 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:29.736206 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:29.737774 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:29.740219 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:29.741357 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:29.741715 (+ 358us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:29.747243 (+ 5528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:29.747253 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:29.747267 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:29.747303 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:29.750866 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:29.751607 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:29.751614 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:29.751616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:29.751745 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:29.752166 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:29.752170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:29.752172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:29.752480 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:29.752706 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:29.752934 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 22:09:29.753214 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11019,"thread_start_us":10848,"threads_started":1}
I20260927 22:09:29.754806 6855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:29.838722 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 6732
I20260927 22:09:29.856087 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38917
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:35323
--webserver_port=43067
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:35323,127.0.97.61:39403
--tserver_master_addrs=127.0.97.62:35323,127.0.97.61:39403,127.0.97.60:40003 with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:30.185133 7527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:30.185431 7527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:30.185595 7527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:30.192921 7527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:30.193079 7527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:30.193251 7527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:09:30.203332 7527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43067
--tserver_master_addrs=127.0.97.62:35323,127.0.97.61:39403,127.0.97.60:40003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:30.204855 7527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:30.206298 7527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:30.219699 7533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:30.219640 7535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:30.220188 7532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:30.220770 7527 server_base.cc:1034] running on GCE node
I20260927 22:09:30.221444 7527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:30.222363 7527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:30.223626 7527 hybrid_clock.cc:648] HybridClock initialized: now 1790546970223520 us; error 87 us; skew 500 ppm
I20260927 22:09:30.224014 7527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:30 dist-test-slave-mp03 krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546970, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:30.227252 7527 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 22:09:30.228780 7527 webserver.cc:460] Webserver started at http://127.0.97.2:43067/ using document root <none> and password file <none>
I20260927 22:09:30.229501 7527 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:30.229650 7527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:30.234812 7527 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:09:30.237334 7542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:30.238830 7527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:09:30.238957 7527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "ebdd20bf62754f138e1d99f4509ba2bd"
format_stamp: "Formatted at 2026-09-27 22:09:27 on dist-test-slave-mp03"
server_key: "4847c3d6318553c4f341b31b4b071b0c"
server_key_iv: "dadc3ec9d72c794ed871564e2571cf75"
server_key_version: "encryptionkey@0"
I20260927 22:09:30.239840 7527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:30.267108 7527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:30.271064 7527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:30.271389 7527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:30.272555 7527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:30.274205 7527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:30.274314 7527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:30.274408 7527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:30.274477 7527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:30.310372 7527 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38917
I20260927 22:09:30.310460 7654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38917 every 8 connection(s)
I20260927 22:09:30.311834 7527 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:09:30.316383 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 7527
I20260927 22:09:30.316716 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 6865
Sep 27 22:09:30 dist-test-slave-mp03 krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546970, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 27 22:09:30 dist-test-slave-mp03 krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546970, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 27 22:09:30 dist-test-slave-mp03 krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546970, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 27 22:09:30 dist-test-slave-mp03 krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260927 22:09:30.349064 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:30.316284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40003 (local address 127.0.97.2:53195)
0927 22:09:30.324689 (+ 8405us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:30.324708 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:30.324821 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:30.329208 (+ 4387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:30.329212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:30.329845 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:30.330133 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:30.330145 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.331699 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.331706 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:30.333651 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:30.333661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.335820 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.335825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:30.336035 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:30.336807 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:30.336833 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:30.339206 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:30.344187 (+ 4981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:30.344196 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:30.344221 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:30.344638 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:30.345126 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:30.345131 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:30.345135 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:30.345399 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:30.346162 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:30.346171 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:30.346661 (+ 490us) client_negotiation.cc:769] Sending connection context
0927 22:09:30.347003 (+ 342us) client_negotiation.cc:240] Negotiation successful
0927 22:09:30.347558 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8285,"mutex_wait_us":371,"thread_start_us":8092,"threads_started":1}
I20260927 22:09:30.350549 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:30.315028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53195 (local address 127.0.97.60:40003)
0927 22:09:30.328734 (+ 13706us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:30.328740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:30.328760 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:09:30.328816 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:30.328820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:30.328906 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:30.329015 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:30.330316 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.331482 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.334934 (+ 3452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.335582 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.335703 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:30.339429 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:30.339435 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:30.339441 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:30.339479 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:30.344019 (+ 4540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:30.344792 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:30.344800 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:30.344805 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:30.344964 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:30.345545 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:30.345553 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:30.345557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:30.345940 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:30.346075 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:30.349837 (+ 3762us) server_negotiation.cc:299] Negotiation successful
0927 22:09:30.350049 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13422,"thread_start_us":13565,"threads_started":1}
I20260927 22:09:30.352171 7655 heartbeater.cc:347] Connected to a master server at 127.0.97.60:40003
I20260927 22:09:30.352510 7655 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:30.353238 7666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:30.323872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39403 (local address 127.0.97.2:39179)
0927 22:09:30.334188 (+ 10316us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:30.334220 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:30.334333 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:30.335902 (+ 1569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:30.335906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:30.336538 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:30.336865 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:30.336880 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.338237 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.338241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:30.339783 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:30.339796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.340760 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.340773 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:30.341008 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:30.341661 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:30.341678 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:30.344368 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:30.349286 (+ 4918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:30.349294 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:30.349300 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:30.349868 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:30.350350 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:30.350356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:30.350359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:30.351483 (+ 1124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:30.352217 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:30.352222 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:30.352376 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 22:09:30.352529 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 22:09:30.352814 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10179,"thread_start_us":10033,"threads_started":1}
I20260927 22:09:30.353418 7655 heartbeater.cc:511] Master 127.0.97.60:40003 requested a full tablet report, sending...
I20260927 22:09:30.353293 7665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:30.322507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39179 (local address 127.0.97.61:39403)
0927 22:09:30.323829 (+ 1322us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:30.323835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:30.334445 (+ 10610us) server_negotiation.cc:407] Connection header received
0927 22:09:30.334528 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:30.334532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:30.334666 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:30.334786 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:30.337056 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.338079 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.339934 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.340548 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.340683 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:30.344587 (+ 3904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:30.344594 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:30.344598 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:30.344638 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:30.349129 (+ 4491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:30.350005 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:30.350010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:30.350013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:30.350181 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:30.351601 (+ 1420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:30.351605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:30.351609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:30.351950 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:30.352121 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:30.352709 (+ 588us) server_negotiation.cc:299] Negotiation successful
0927 22:09:30.352938 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1207,"thread_start_us":1101,"threads_started":1}
I20260927 22:09:30.354655 7657 heartbeater.cc:347] Connected to a master server at 127.0.97.61:39403
I20260927 22:09:30.354719 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:30.317808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50142 (local address 127.0.97.3:38105)
0927 22:09:30.322191 (+ 4383us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:30.322203 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:30.326204 (+ 4001us) server_negotiation.cc:407] Connection header received
0927 22:09:30.326307 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:30.326320 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:30.326683 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:30.326852 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:30.334510 (+ 7658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.336036 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.338525 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.339405 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.339504 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:30.346819 (+ 7315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:30.346834 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:30.346852 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:30.346891 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:30.351087 (+ 4196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:30.351988 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:30.351997 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:30.352000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:30.352173 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:30.352725 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:30.352730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:30.352733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:30.353107 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:30.353449 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:30.354030 (+ 581us) server_negotiation.cc:299] Negotiation successful
0927 22:09:30.354275 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4166,"thread_start_us":4944,"threads_started":1}
I20260927 22:09:30.354785 7657 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:30.355185 7334 ts_manager.cc:194] Registered new tserver with Master: ebdd20bf62754f138e1d99f4509ba2bd (127.0.97.2:38917)
I20260927 22:09:30.355230 7657 heartbeater.cc:511] Master 127.0.97.61:39403 requested a full tablet report, sending...
I20260927 22:09:30.356097 6988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:30.356437 7144 ts_manager.cc:194] Registered new tserver with Master: ebdd20bf62754f138e1d99f4509ba2bd (127.0.97.2:38917)
I20260927 22:09:30.357491 7659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:30.314767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41021 (local address 127.0.97.62:35323)
0927 22:09:30.315180 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:30.315189 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:30.318085 (+ 2896us) server_negotiation.cc:407] Connection header received
0927 22:09:30.322367 (+ 4282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:30.322377 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:30.322496 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:30.322632 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:30.327168 (+ 4536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.328608 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.333441 (+ 4833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.334143 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.334277 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:30.346008 (+ 11731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:30.346015 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:30.346019 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:30.346059 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:30.349865 (+ 3806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:30.350994 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:30.351000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:30.351003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:30.351146 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:30.351604 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:30.351612 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:30.351620 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:30.351956 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:30.352083 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:30.356638 (+ 4555us) server_negotiation.cc:299] Negotiation successful
0927 22:09:30.356945 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":397,"threads_started":1}
I20260927 22:09:30.359409 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:30.315614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.97.2:41021)
0927 22:09:30.317879 (+ 2265us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:30.317936 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:30.321331 (+ 3395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:30.323341 (+ 2010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:30.323360 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:30.324328 (+ 968us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:30.325256 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:30.325278 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.328812 (+ 3534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.328817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:30.333283 (+ 4466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:30.333294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.337981 (+ 4687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.337987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:30.338152 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:30.339778 (+ 1626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:30.339798 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:30.345759 (+ 5961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:30.350012 (+ 4253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:30.350017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:30.350020 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:30.350857 (+ 837us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:30.351305 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:30.351312 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:30.351315 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:30.351462 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:30.356125 (+ 4663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:30.356129 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:30.356293 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 22:09:30.357911 (+ 1618us) client_negotiation.cc:240] Negotiation successful
0927 22:09:30.358619 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1874,"mutex_wait_us":5,"thread_start_us":4430,"threads_started":1}
I20260927 22:09:30.360332 7656 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35323
I20260927 22:09:30.360499 7656 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:30.361020 7656 heartbeater.cc:511] Master 127.0.97.62:35323 requested a full tablet report, sending...
I20260927 22:09:30.361984 6536 ts_manager.cc:194] Re-registered known tserver with Master: ebdd20bf62754f138e1d99f4509ba2bd (127.0.97.2:38917)
I20260927 22:09:30.362895 6536 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41021
I20260927 22:09:30.443869 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 6865
I20260927 22:09:30.460892 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:38105
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:35323
--webserver_port=32909
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:35323,127.0.97.61:39403
--tserver_master_addrs=127.0.97.62:35323,127.0.97.61:39403,127.0.97.60:40003 with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:09:30.749662 7516 heartbeater.cc:503] Master 127.0.97.62:35323 was elected leader, sending a full tablet report...
W20260927 22:09:30.788703 7668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:30.788969 7668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:30.789176 7668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:30.796370 7668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:30.796476 7668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:30.796653 7668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:09:30.806768 7668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:38105
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=32909
--tserver_master_addrs=127.0.97.62:35323,127.0.97.61:39403,127.0.97.60:40003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:30.808210 7668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:30.809499 7668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:30.822167 7673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:30.822217 7676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:30.822240 7674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:30.826694 7668 server_base.cc:1034] running on GCE node
I20260927 22:09:30.827360 7668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:30.828387 7668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:30.829609 7668 hybrid_clock.cc:648] HybridClock initialized: now 1790546970829557 us; error 49 us; skew 500 ppm
I20260927 22:09:30.829948 7668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:30 dist-test-slave-mp03 krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546970, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:30.833137 7668 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 22:09:30.834545 7668 webserver.cc:460] Webserver started at http://127.0.97.3:32909/ using document root <none> and password file <none>
I20260927 22:09:30.835369 7668 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:30.835482 7668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:30.840881 7668 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260927 22:09:30.843655 7683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:30.845103 7668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 22:09:30.845260 7668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "0f99af01ae61446fb3569c69c2e81936"
format_stamp: "Formatted at 2026-09-27 22:09:27 on dist-test-slave-mp03"
server_key: "2fdcb8a665969ad275db30add421d178"
server_key_iv: "b774dadebfe4685f3d3d016773824c4b"
server_key_version: "encryptionkey@0"
I20260927 22:09:30.846264 7668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:30.859123 7668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:30.863477 7668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:30.863754 7668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:30.864660 7668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:30.865931 7668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:30.866060 7668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:30.866178 7668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:30.866262 7668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:30.898391 7668 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38105
I20260927 22:09:30.898480 7795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38105 every 8 connection(s)
I20260927 22:09:30.899843 7668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:09:30.901326 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 7668
Sep 27 22:09:30 dist-test-slave-mp03 krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546970, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 27 22:09:30 dist-test-slave-mp03 krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546970, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 27 22:09:30 dist-test-slave-mp03 krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546970, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 22:09:30.966568 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:30.932339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55580 (local address 127.0.97.62:35323)
0927 22:09:30.939411 (+ 7072us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:30.939420 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:30.945938 (+ 6518us) server_negotiation.cc:407] Connection header received
0927 22:09:30.945994 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:30.945998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:30.946082 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:30.946174 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:30.948929 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.950314 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.953481 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.954174 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.954695 (+ 521us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:30.958457 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:30.958464 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:30.958469 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:30.958507 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:30.962110 (+ 3603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:30.963956 (+ 1846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:30.963962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:30.963966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:30.964104 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:30.964553 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:30.964560 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:30.964567 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:30.964891 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:30.965119 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:30.965971 (+ 852us) server_negotiation.cc:299] Negotiation successful
0927 22:09:30.966198 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6921,"thread_start_us":7946,"threads_started":1}
I20260927 22:09:30.967294 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:30.902802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34619 (local address 127.0.97.60:40003)
0927 22:09:30.930327 (+ 27525us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:30.930333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:30.930357 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:09:30.935267 (+ 4910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:30.935276 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:30.935391 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:30.935517 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:30.936833 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.938213 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.941896 (+ 3683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.943382 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.943539 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:30.949859 (+ 6320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:30.949867 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:30.949875 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:30.949928 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:30.960397 (+ 10469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:30.961705 (+ 1308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:30.961715 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:30.961721 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:30.961900 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:30.962559 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:30.962567 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:30.962573 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:30.963051 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:30.963227 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:30.966078 (+ 2851us) server_negotiation.cc:299] Negotiation successful
0927 22:09:30.966376 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27405,"thread_start_us":27277,"threads_started":1}
I20260927 22:09:30.969089 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:30.902823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44513 (local address 127.0.97.62:35323)
0927 22:09:30.929984 (+ 27161us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:30.929991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:30.931107 (+ 1116us) server_negotiation.cc:407] Connection header received
0927 22:09:30.931486 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:30.931491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:30.931584 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:30.931702 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:30.935121 (+ 3419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.936531 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.940711 (+ 4180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.941537 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.941684 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:30.950858 (+ 9174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:30.950865 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:30.950869 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:30.950907 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:30.963768 (+ 12861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:30.965592 (+ 1824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:30.965598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:30.965602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:30.965719 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:30.966874 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:30.966880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:30.966884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:30.967205 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:30.967331 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:30.968264 (+ 933us) server_negotiation.cc:299] Negotiation successful
0927 22:09:30.968574 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26999,"thread_start_us":27549,"threads_started":1}
I20260927 22:09:30.969535 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:30.904223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.97.3:44513)
0927 22:09:30.930977 (+ 26754us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:30.930997 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:30.931157 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:30.932625 (+ 1468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:30.932642 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:30.933812 (+ 1170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:30.934892 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:30.934923 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.936868 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.936876 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:30.940094 (+ 3218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:30.940114 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.941793 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.941803 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:30.942172 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:30.943264 (+ 1092us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:30.943343 (+ 79us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:30.949807 (+ 6464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:30.963884 (+ 14077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:30.963890 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:30.963895 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:30.964463 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:30.965827 (+ 1364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:30.965833 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:30.965838 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:30.966011 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:30.967781 (+ 1770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:30.967786 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:30.967918 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 22:09:30.968062 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 22:09:30.968335 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26629,"mutex_wait_us":459,"thread_start_us":33979,"threads_started":1}
I20260927 22:09:30.971271 7797 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35323
I20260927 22:09:30.971679 7797 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:30.972733 7797 heartbeater.cc:511] Master 127.0.97.62:35323 requested a full tablet report, sending...
I20260927 22:09:30.973785 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:30.940548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39403 (local address 127.0.97.3:54283)
0927 22:09:30.943606 (+ 3058us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:30.943629 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:30.943742 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:30.944473 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:30.944478 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:30.945030 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:30.945453 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:30.945467 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.947306 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.947314 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:30.948931 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:30.948944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.952976 (+ 4032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.952982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:30.953158 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:30.955614 (+ 2456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:30.955641 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:30.959769 (+ 4128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:30.967200 (+ 7431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:30.967205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:30.967208 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:30.967674 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:30.970286 (+ 2612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:30.970292 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:30.970298 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:30.970468 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:30.972780 (+ 2312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:30.972784 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:30.972932 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 22:09:30.973094 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 22:09:30.973374 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2926,"thread_start_us":3288,"threads_started":1}
I20260927 22:09:30.974715 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:30.938782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54283 (local address 127.0.97.61:39403)
0927 22:09:30.944150 (+ 5368us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:30.944156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:30.944173 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:09:30.944229 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:30.944235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:30.944307 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:30.944749 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:30.945863 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.946957 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.949999 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.952640 (+ 2641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.952752 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:30.960003 (+ 7251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:30.960010 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:30.960018 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:30.960068 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:30.966443 (+ 6375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:30.969987 (+ 3544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:30.969994 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:30.970000 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:30.970150 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:30.972326 (+ 2176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:30.972331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:30.972337 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:30.972644 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:30.973980 (+ 1336us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:30.974107 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 22:09:30.974325 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5253,"thread_start_us":4236,"threads_started":1}
I20260927 22:09:30.975147 6536 ts_manager.cc:194] Re-registered known tserver with Master: 0f99af01ae61446fb3569c69c2e81936 (127.0.97.3:38105)
I20260927 22:09:30.976441 6536 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44513
I20260927 22:09:30.976622 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:30.903615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40003 (local address 127.0.97.3:34619)
0927 22:09:30.927460 (+ 23845us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:30.927534 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:30.930381 (+ 2847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:30.935748 (+ 5367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:30.935755 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:30.936310 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:30.936648 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:30.936662 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.938420 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.938428 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:30.940508 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:30.940528 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.943923 (+ 3395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.943932 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:30.944179 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:30.945032 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:30.945061 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:30.949469 (+ 4408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:30.960663 (+ 11194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:30.960677 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:30.960704 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:30.961473 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:30.962050 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:30.962058 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:30.962064 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:30.962388 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:30.963316 (+ 928us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:30.963329 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:30.963944 (+ 615us) client_negotiation.cc:769] Sending connection context
0927 22:09:30.964333 (+ 389us) client_negotiation.cc:240] Negotiation successful
0927 22:09:30.965071 (+ 738us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23396,"mutex_wait_us":1,"thread_start_us":27732,"threads_started":1}
I20260927 22:09:30.978082 7798 heartbeater.cc:347] Connected to a master server at 127.0.97.61:39403
I20260927 22:09:30.978204 7798 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:30.978682 7798 heartbeater.cc:511] Master 127.0.97.61:39403 requested a full tablet report, sending...
I20260927 22:09:30.979238 7796 heartbeater.cc:347] Connected to a master server at 127.0.97.60:40003
I20260927 22:09:30.979346 7796 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:30.979708 7796 heartbeater.cc:511] Master 127.0.97.60:40003 requested a full tablet report, sending...
I20260927 22:09:30.979873 7144 ts_manager.cc:194] Registered new tserver with Master: 0f99af01ae61446fb3569c69c2e81936 (127.0.97.3:38105)
I20260927 22:09:30.980901 7334 ts_manager.cc:194] Registered new tserver with Master: 0f99af01ae61446fb3569c69c2e81936 (127.0.97.3:38105)
I20260927 22:09:30.986526 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:30.951371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48260 (local address 127.0.97.61:39403)
0927 22:09:30.968394 (+ 17023us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:30.968401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:30.968418 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:09:30.968467 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:30.968473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:30.968541 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:30.968636 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:30.971589 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.972773 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.974554 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.975353 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.975469 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:30.979047 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:30.979053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:30.979057 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:30.979099 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:30.983671 (+ 4572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:30.984554 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:30.984559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:30.984564 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:30.984706 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:30.985088 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:30.985093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:30.985096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:30.985417 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:30.985632 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:30.985913 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 22:09:30.986109 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16905,"thread_start_us":4266,"threads_started":1}
I20260927 22:09:30.987450 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:30.932950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42122 (local address 127.0.97.60:40003)
0927 22:09:30.946687 (+ 13737us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:30.946693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:30.951415 (+ 4722us) server_negotiation.cc:407] Connection header received
0927 22:09:30.951464 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:30.951470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:30.951564 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:30.951659 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:30.957763 (+ 6104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.958951 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.970706 (+ 11755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:30.971308 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:30.971418 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:30.979020 (+ 7602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:30.979031 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:30.979035 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:30.979075 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:30.983120 (+ 4045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:30.983874 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:30.983879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:30.983882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:30.984063 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:30.984576 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:30.984632 (+ 56us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:30.984635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:30.984954 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:30.985175 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:30.986404 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0927 22:09:30.986658 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13599,"thread_start_us":13489,"threads_started":1}
I20260927 22:09:30.982453 6537 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:09:30.991860 6537 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:09:31.048462 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:31.013295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47946 (local address 127.0.97.2:38917)
0927 22:09:31.027292 (+ 13997us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:31.027303 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:31.035746 (+ 8443us) server_negotiation.cc:407] Connection header received
0927 22:09:31.035863 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:31.035872 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:31.036183 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:31.036414 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:09:31.037896 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.039935 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.044626 (+ 4691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.047281 (+ 2655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.047402 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:31.047630 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:31.047991 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 22:09:31.048171 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13813,"thread_start_us":13990,"threads_started":1}
I20260927 22:09:31.048489 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:31.012983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38917 (local address 127.0.0.1:47946)
0927 22:09:31.035446 (+ 22463us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:31.035466 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:31.035600 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:31.037145 (+ 1545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:31.037153 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:31.037190 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:09:31.037674 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:31.037689 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.040153 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.040159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:31.042810 (+ 2651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:31.044408 (+ 1598us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.047557 (+ 3149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.047567 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:31.047791 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:31.047835 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 22:09:31.047935 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 22:09:31.048073 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22284,"mutex_wait_us":1,"thread_start_us":22113,"threads_started":1}
I20260927 22:09:31.052542 7590 tablet_service.cc:1467] Processing CreateTablet for tablet a294090772894adf95121fe922ddfc45 (DEFAULT_TABLE table=test-table [id=e9ab1e9591904b20a6df05d266e77aac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:09:31.052703 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:31.035589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43179 (local address 127.0.0.1:51286)
0927 22:09:31.042258 (+ 6669us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:31.042287 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:31.042669 (+ 382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:31.044140 (+ 1471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:31.044148 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:31.044183 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:09:31.044716 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:31.044731 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.047008 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.047016 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:31.049696 (+ 2680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:31.049711 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.051839 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.051845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:31.052025 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:31.052059 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:09:31.052149 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 22:09:31.052275 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6481,"mutex_wait_us":1,"thread_start_us":6724,"threads_started":1}
I20260927 22:09:31.054023 7590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a294090772894adf95121fe922ddfc45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:31.054953 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:31.035784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51286 (local address 127.0.97.1:43179)
0927 22:09:31.043327 (+ 7543us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:31.043352 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:31.043388 (+ 36us) server_negotiation.cc:407] Connection header received
0927 22:09:31.043513 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:31.043524 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:31.043880 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:31.044158 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:09:31.045034 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.046876 (+ 1842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.049890 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.051357 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.051483 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:31.051681 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:31.054458 (+ 2777us) server_negotiation.cc:299] Negotiation successful
0927 22:09:31.054657 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7322,"thread_start_us":7213,"threads_started":1}
I20260927 22:09:31.056162 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:31.011410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38105 (local address 127.0.0.1:50146)
0927 22:09:31.029551 (+ 18141us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:31.029577 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:31.029741 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:31.047179 (+ 17438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:31.047184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:31.047219 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:09:31.047718 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:31.047736 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.051190 (+ 3454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.051195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:31.053393 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:31.053407 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.055297 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.055303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:31.055584 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:31.055628 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 22:09:31.055691 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 22:09:31.055832 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17964,"thread_start_us":17799,"threads_started":1}
I20260927 22:09:31.057090 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:31.011617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50146 (local address 127.0.97.3:38105)
0927 22:09:31.046531 (+ 34914us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:31.046543 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:31.046580 (+ 37us) server_negotiation.cc:407] Connection header received
0927 22:09:31.046703 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:31.046709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:31.046978 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:31.047177 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:09:31.048827 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.050932 (+ 2105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.053567 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.055008 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.056404 (+ 1396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:31.056583 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:31.056641 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 22:09:31.056786 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34698,"thread_start_us":34531,"threads_started":1}
I20260927 22:09:31.058820 7450 tablet_service.cc:1467] Processing CreateTablet for tablet a294090772894adf95121fe922ddfc45 (DEFAULT_TABLE table=test-table [id=e9ab1e9591904b20a6df05d266e77aac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:09:31.060215 7450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a294090772894adf95121fe922ddfc45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:31.060813 7731 tablet_service.cc:1467] Processing CreateTablet for tablet a294090772894adf95121fe922ddfc45 (DEFAULT_TABLE table=test-table [id=e9ab1e9591904b20a6df05d266e77aac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:09:31.062093 7731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a294090772894adf95121fe922ddfc45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:31.067191 7833 tablet_bootstrap.cc:492] T a294090772894adf95121fe922ddfc45 P ebdd20bf62754f138e1d99f4509ba2bd: Bootstrap starting.
I20260927 22:09:31.070112 7833 tablet_bootstrap.cc:654] T a294090772894adf95121fe922ddfc45 P ebdd20bf62754f138e1d99f4509ba2bd: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:31.071650 7833 log.cc:824] T a294090772894adf95121fe922ddfc45 P ebdd20bf62754f138e1d99f4509ba2bd: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:31.076259 7833 tablet_bootstrap.cc:492] T a294090772894adf95121fe922ddfc45 P ebdd20bf62754f138e1d99f4509ba2bd: No bootstrap required, opened a new log
I20260927 22:09:31.076570 7833 ts_tablet_manager.cc:1397] T a294090772894adf95121fe922ddfc45 P ebdd20bf62754f138e1d99f4509ba2bd: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260927 22:09:31.080295 7835 tablet_bootstrap.cc:492] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693: Bootstrap starting.
I20260927 22:09:31.084210 7835 tablet_bootstrap.cc:654] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:31.085073 7836 tablet_bootstrap.cc:492] T a294090772894adf95121fe922ddfc45 P 0f99af01ae61446fb3569c69c2e81936: Bootstrap starting.
I20260927 22:09:31.086540 7835 log.cc:824] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:31.088565 7836 tablet_bootstrap.cc:654] T a294090772894adf95121fe922ddfc45 P 0f99af01ae61446fb3569c69c2e81936: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:31.089481 7836 log.cc:824] T a294090772894adf95121fe922ddfc45 P 0f99af01ae61446fb3569c69c2e81936: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:31.089748 7835 tablet_bootstrap.cc:492] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693: No bootstrap required, opened a new log
I20260927 22:09:31.090032 7835 ts_tablet_manager.cc:1397] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 22:09:31.091923 7833 raft_consensus.cc:348] T a294090772894adf95121fe922ddfc45 P ebdd20bf62754f138e1d99f4509ba2bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f99af01ae61446fb3569c69c2e81936" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38105 } } peers { permanent_uuid: "ebdd20bf62754f138e1d99f4509ba2bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38917 } } peers { permanent_uuid: "100d5e2a45074ac29b114b3bf0e3f693" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43179 } }
I20260927 22:09:31.092373 7833 raft_consensus.cc:374] T a294090772894adf95121fe922ddfc45 P ebdd20bf62754f138e1d99f4509ba2bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:31.092468 7833 raft_consensus.cc:729] T a294090772894adf95121fe922ddfc45 P ebdd20bf62754f138e1d99f4509ba2bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebdd20bf62754f138e1d99f4509ba2bd, State: Initialized, Role: FOLLOWER
I20260927 22:09:31.092968 7836 tablet_bootstrap.cc:492] T a294090772894adf95121fe922ddfc45 P 0f99af01ae61446fb3569c69c2e81936: No bootstrap required, opened a new log
I20260927 22:09:31.093209 7836 ts_tablet_manager.cc:1397] T a294090772894adf95121fe922ddfc45 P 0f99af01ae61446fb3569c69c2e81936: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260927 22:09:31.093091 7833 consensus_queue.cc:260] T a294090772894adf95121fe922ddfc45 P ebdd20bf62754f138e1d99f4509ba2bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f99af01ae61446fb3569c69c2e81936" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38105 } } peers { permanent_uuid: "ebdd20bf62754f138e1d99f4509ba2bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38917 } } peers { permanent_uuid: "100d5e2a45074ac29b114b3bf0e3f693" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43179 } }
I20260927 22:09:31.094803 7833 ts_tablet_manager.cc:1428] T a294090772894adf95121fe922ddfc45 P ebdd20bf62754f138e1d99f4509ba2bd: Time spent starting tablet: real 0.018s user 0.013s sys 0.003s
I20260927 22:09:31.095396 7656 heartbeater.cc:503] Master 127.0.97.62:35323 was elected leader, sending a full tablet report...
I20260927 22:09:31.102391 7835 raft_consensus.cc:348] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f99af01ae61446fb3569c69c2e81936" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38105 } } peers { permanent_uuid: "ebdd20bf62754f138e1d99f4509ba2bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38917 } } peers { permanent_uuid: "100d5e2a45074ac29b114b3bf0e3f693" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43179 } }
I20260927 22:09:31.102926 7835 raft_consensus.cc:374] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:31.103065 7835 raft_consensus.cc:729] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100d5e2a45074ac29b114b3bf0e3f693, State: Initialized, Role: FOLLOWER
I20260927 22:09:31.103699 7835 consensus_queue.cc:260] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f99af01ae61446fb3569c69c2e81936" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38105 } } peers { permanent_uuid: "ebdd20bf62754f138e1d99f4509ba2bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38917 } } peers { permanent_uuid: "100d5e2a45074ac29b114b3bf0e3f693" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43179 } }
I20260927 22:09:31.105334 7835 ts_tablet_manager.cc:1428] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260927 22:09:31.105182 7836 raft_consensus.cc:348] T a294090772894adf95121fe922ddfc45 P 0f99af01ae61446fb3569c69c2e81936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f99af01ae61446fb3569c69c2e81936" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38105 } } peers { permanent_uuid: "ebdd20bf62754f138e1d99f4509ba2bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38917 } } peers { permanent_uuid: "100d5e2a45074ac29b114b3bf0e3f693" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43179 } }
I20260927 22:09:31.105638 7836 raft_consensus.cc:374] T a294090772894adf95121fe922ddfc45 P 0f99af01ae61446fb3569c69c2e81936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:31.105742 7836 raft_consensus.cc:729] T a294090772894adf95121fe922ddfc45 P 0f99af01ae61446fb3569c69c2e81936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f99af01ae61446fb3569c69c2e81936, State: Initialized, Role: FOLLOWER
I20260927 22:09:31.106335 7836 consensus_queue.cc:260] T a294090772894adf95121fe922ddfc45 P 0f99af01ae61446fb3569c69c2e81936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f99af01ae61446fb3569c69c2e81936" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38105 } } peers { permanent_uuid: "ebdd20bf62754f138e1d99f4509ba2bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38917 } } peers { permanent_uuid: "100d5e2a45074ac29b114b3bf0e3f693" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43179 } }
I20260927 22:09:31.108968 7836 ts_tablet_manager.cc:1428] T a294090772894adf95121fe922ddfc45 P 0f99af01ae61446fb3569c69c2e81936: Time spent starting tablet: real 0.016s user 0.013s sys 0.001s
I20260927 22:09:31.109404 7797 heartbeater.cc:503] Master 127.0.97.62:35323 was elected leader, sending a full tablet report...
W20260927 22:09:31.176460 7804 tablet.cc:2369] T a294090772894adf95121fe922ddfc45 P 0f99af01ae61446fb3569c69c2e81936: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:09:31.208263 7518 tablet.cc:2369] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:09:31.220638 7840 raft_consensus.cc:482] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:09:31.220949 7840 raft_consensus.cc:504] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f99af01ae61446fb3569c69c2e81936" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38105 } } peers { permanent_uuid: "ebdd20bf62754f138e1d99f4509ba2bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38917 } } peers { permanent_uuid: "100d5e2a45074ac29b114b3bf0e3f693" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43179 } }
I20260927 22:09:31.222651 7840 leader_election.cc:288] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f99af01ae61446fb3569c69c2e81936 (127.0.97.3:38105), ebdd20bf62754f138e1d99f4509ba2bd (127.0.97.2:38917)
I20260927 22:09:31.230084 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:31.223148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40925 (local address 127.0.97.2:38917)
0927 22:09:31.223342 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:31.223350 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:31.224253 (+ 903us) server_negotiation.cc:407] Connection header received
0927 22:09:31.224480 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:31.224488 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:31.224575 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:31.224694 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:09:31.225283 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.226268 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.228284 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.229113 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.229256 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:31.229363 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:31.229681 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 22:09:31.229803 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:09:31.230207 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:31.223120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56637 (local address 127.0.97.3:38105)
0927 22:09:31.223316 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:31.223322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:31.224818 (+ 1496us) server_negotiation.cc:407] Connection header received
0927 22:09:31.225001 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:31.225007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:31.225087 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:31.225188 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:09:31.225827 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.226638 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.228486 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.229316 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.229486 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:31.229588 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:31.229843 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 22:09:31.229969 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:09:31.230374 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:31.222919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38105 (local address 127.0.97.1:56637)
0927 22:09:31.224707 (+ 1788us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:31.224724 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:31.224870 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:31.225273 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:31.225278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:31.225304 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:09:31.225625 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:31.225671 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.226774 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.226779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:31.228315 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:31.228326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.229546 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.229552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:31.229696 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:31.229718 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 22:09:31.229820 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 22:09:31.229940 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1646,"mutex_wait_us":19,"spinlock_wait_cycles":4864,"thread_start_us":1520,"threads_started":1}
I20260927 22:09:31.230747 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:31.222877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38917 (local address 127.0.97.1:40925)
0927 22:09:31.224067 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:31.224094 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:31.224294 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:31.224709 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:31.224713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:31.224733 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:09:31.225106 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:31.225119 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.226401 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.226409 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:31.228121 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:31.228132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.229309 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.229316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:31.229502 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:31.229543 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 22:09:31.229645 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 22:09:31.229778 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"mutex_wait_us":43,"thread_start_us":325,"threads_started":1}
I20260927 22:09:31.231397 7751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a294090772894adf95121fe922ddfc45" candidate_uuid: "100d5e2a45074ac29b114b3bf0e3f693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f99af01ae61446fb3569c69c2e81936" is_pre_election: true
I20260927 22:09:31.231624 7610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a294090772894adf95121fe922ddfc45" candidate_uuid: "100d5e2a45074ac29b114b3bf0e3f693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebdd20bf62754f138e1d99f4509ba2bd" is_pre_election: true
I20260927 22:09:31.231904 7751 raft_consensus.cc:2454] T a294090772894adf95121fe922ddfc45 P 0f99af01ae61446fb3569c69c2e81936 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 100d5e2a45074ac29b114b3bf0e3f693 in term 0.
I20260927 22:09:31.232139 7610 raft_consensus.cc:2454] T a294090772894adf95121fe922ddfc45 P ebdd20bf62754f138e1d99f4509ba2bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 100d5e2a45074ac29b114b3bf0e3f693 in term 0.
I20260927 22:09:31.232584 7403 leader_election.cc:302] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f99af01ae61446fb3569c69c2e81936, 100d5e2a45074ac29b114b3bf0e3f693; no voters:
I20260927 22:09:31.232962 7840 raft_consensus.cc:2781] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:09:31.233139 7840 raft_consensus.cc:482] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:09:31.233319 7840 raft_consensus.cc:3037] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:31.236042 7840 raft_consensus.cc:504] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f99af01ae61446fb3569c69c2e81936" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38105 } } peers { permanent_uuid: "ebdd20bf62754f138e1d99f4509ba2bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38917 } } peers { permanent_uuid: "100d5e2a45074ac29b114b3bf0e3f693" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43179 } }
I20260927 22:09:31.236620 7840 leader_election.cc:288] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693 [CANDIDATE]: Term 1 election: Requested vote from peers 0f99af01ae61446fb3569c69c2e81936 (127.0.97.3:38105), ebdd20bf62754f138e1d99f4509ba2bd (127.0.97.2:38917)
I20260927 22:09:31.237135 7610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a294090772894adf95121fe922ddfc45" candidate_uuid: "100d5e2a45074ac29b114b3bf0e3f693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebdd20bf62754f138e1d99f4509ba2bd"
I20260927 22:09:31.237325 7610 raft_consensus.cc:3037] T a294090772894adf95121fe922ddfc45 P ebdd20bf62754f138e1d99f4509ba2bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:31.237339 7751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a294090772894adf95121fe922ddfc45" candidate_uuid: "100d5e2a45074ac29b114b3bf0e3f693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f99af01ae61446fb3569c69c2e81936"
I20260927 22:09:31.237547 7751 raft_consensus.cc:3037] T a294090772894adf95121fe922ddfc45 P 0f99af01ae61446fb3569c69c2e81936 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:31.239593 7610 raft_consensus.cc:2454] T a294090772894adf95121fe922ddfc45 P ebdd20bf62754f138e1d99f4509ba2bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 100d5e2a45074ac29b114b3bf0e3f693 in term 1.
I20260927 22:09:31.239746 7751 raft_consensus.cc:2454] T a294090772894adf95121fe922ddfc45 P 0f99af01ae61446fb3569c69c2e81936 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 100d5e2a45074ac29b114b3bf0e3f693 in term 1.
I20260927 22:09:31.240190 7406 leader_election.cc:302] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0f99af01ae61446fb3569c69c2e81936, 100d5e2a45074ac29b114b3bf0e3f693, ebdd20bf62754f138e1d99f4509ba2bd; no voters:
I20260927 22:09:31.240741 7840 raft_consensus.cc:2781] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:09:31.241199 7840 raft_consensus.cc:686] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693 [term 1 LEADER]: Becoming Leader. State: Replica: 100d5e2a45074ac29b114b3bf0e3f693, State: Running, Role: LEADER
I20260927 22:09:31.241798 7840 consensus_queue.cc:237] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f99af01ae61446fb3569c69c2e81936" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38105 } } peers { permanent_uuid: "ebdd20bf62754f138e1d99f4509ba2bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38917 } } peers { permanent_uuid: "100d5e2a45074ac29b114b3bf0e3f693" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43179 } }
I20260927 22:09:31.247335 6537 catalog_manager.cc:5495] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693 reported cstate change: term changed from 0 to 1, leader changed from <none> to 100d5e2a45074ac29b114b3bf0e3f693 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "100d5e2a45074ac29b114b3bf0e3f693" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f99af01ae61446fb3569c69c2e81936" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebdd20bf62754f138e1d99f4509ba2bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "100d5e2a45074ac29b114b3bf0e3f693" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43179 } health_report { overall_health: HEALTHY } } }
I20260927 22:09:31.270989 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:31.265945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51288 (local address 127.0.97.1:43179)
0927 22:09:31.266163 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:31.266167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:31.266182 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:09:31.266260 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:31.266263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:31.266343 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:31.266496 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:09:31.266901 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.267589 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.268850 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.269264 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.269411 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:31.269883 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:09:31.270401 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:09:31.270561 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:31.270642 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 22:09:31.270723 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:09:31.280133 7610 raft_consensus.cc:1261] T a294090772894adf95121fe922ddfc45 P ebdd20bf62754f138e1d99f4509ba2bd [term 1 FOLLOWER]: Refusing update from remote peer 100d5e2a45074ac29b114b3bf0e3f693: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:09:31.280133 7751 raft_consensus.cc:1261] T a294090772894adf95121fe922ddfc45 P 0f99af01ae61446fb3569c69c2e81936 [term 1 FOLLOWER]: Refusing update from remote peer 100d5e2a45074ac29b114b3bf0e3f693: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:09:31.281196 7840 consensus_queue.cc:1035] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f99af01ae61446fb3569c69c2e81936" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:09:31.281962 7844 consensus_queue.cc:1035] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebdd20bf62754f138e1d99f4509ba2bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 22:09:31.295914 7849 mvcc.cc:205] Tried to move back new op lower bound from 7334080394353242112 to 7334080394211385344. Current Snapshot: MvccSnapshot[applied={T|T < 7334080394353242112}]
I20260927 22:09:31.300104 7847 mvcc.cc:205] Tried to move back new op lower bound from 7334080394353242112 to 7334080394211385344. Current Snapshot: MvccSnapshot[applied={T|T < 7334080394353242112}]
I20260927 22:09:31.322166 6537 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:09:31.322417 6537 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:09:31.328372 6537 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae6791bd85ae4fceb7d3ac0a20c86d2d: Sending DeleteTablet for 3 replicas of tablet a294090772894adf95121fe922ddfc45
I20260927 22:09:31.329424 7731 tablet_service.cc:1514] Processing DeleteTablet for tablet a294090772894adf95121fe922ddfc45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:31 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50146
I20260927 22:09:31.329929 7590 tablet_service.cc:1514] Processing DeleteTablet for tablet a294090772894adf95121fe922ddfc45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:31 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47946
I20260927 22:09:31.330343 7858 tablet_replica.cc:330] T a294090772894adf95121fe922ddfc45 P 0f99af01ae61446fb3569c69c2e81936: stopping tablet replica
I20260927 22:09:31.330787 7858 raft_consensus.cc:2229] T a294090772894adf95121fe922ddfc45 P 0f99af01ae61446fb3569c69c2e81936 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:09:31.331275 7858 raft_consensus.cc:2258] T a294090772894adf95121fe922ddfc45 P 0f99af01ae61446fb3569c69c2e81936 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:09:31.333753 7450 tablet_service.cc:1514] Processing DeleteTablet for tablet a294090772894adf95121fe922ddfc45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:31 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51286
I20260927 22:09:31.335151 7858 ts_tablet_manager.cc:1905] T a294090772894adf95121fe922ddfc45 P 0f99af01ae61446fb3569c69c2e81936: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:31.336429 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 6505
I20260927 22:09:31.337936 6567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:31.345541 7858 ts_tablet_manager.cc:1918] T a294090772894adf95121fe922ddfc45 P 0f99af01ae61446fb3569c69c2e81936: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:09:31.345708 7858 log.cc:1196] T a294090772894adf95121fe922ddfc45 P 0f99af01ae61446fb3569c69c2e81936: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/a294090772894adf95121fe922ddfc45
I20260927 22:09:31.346344 7858 ts_tablet_manager.cc:1939] T a294090772894adf95121fe922ddfc45 P 0f99af01ae61446fb3569c69c2e81936: Deleting consensus metadata
I20260927 22:09:31.354543 7861 tablet_replica.cc:330] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693: stopping tablet replica
I20260927 22:09:31.355413 7861 raft_consensus.cc:2229] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:09:31.356382 7861 raft_consensus.cc:2258] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:09:31.356954 7859 tablet_replica.cc:330] T a294090772894adf95121fe922ddfc45 P ebdd20bf62754f138e1d99f4509ba2bd: stopping tablet replica
I20260927 22:09:31.357481 7859 raft_consensus.cc:2229] T a294090772894adf95121fe922ddfc45 P ebdd20bf62754f138e1d99f4509ba2bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:09:31.358060 7859 raft_consensus.cc:2258] T a294090772894adf95121fe922ddfc45 P ebdd20bf62754f138e1d99f4509ba2bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:09:31.360600 7859 ts_tablet_manager.cc:1905] T a294090772894adf95121fe922ddfc45 P ebdd20bf62754f138e1d99f4509ba2bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:31.362087 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:31.333099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48276 (local address 127.0.97.61:39403)
0927 22:09:31.333274 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:31.333282 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:31.333302 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:09:31.342233 (+ 8931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:31.342241 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:31.342358 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:31.342489 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:31.345946 (+ 3457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.347395 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.349986 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.350782 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.351013 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:31.355268 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:31.355275 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:31.355280 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:31.355353 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:31.358912 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:31.359782 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:31.359788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:31.359791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:31.359929 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:31.360391 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:31.360397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:31.360400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:31.360729 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:31.360933 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:31.361531 (+ 598us) server_negotiation.cc:299] Negotiation successful
0927 22:09:31.361757 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:09:31.366217 7861 ts_tablet_manager.cc:1905] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:31.374410 7861 ts_tablet_manager.cc:1918] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:09:31.374580 7861 log.cc:1196] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/wal/wals/a294090772894adf95121fe922ddfc45
I20260927 22:09:31.375224 7861 ts_tablet_manager.cc:1939] T a294090772894adf95121fe922ddfc45 P 100d5e2a45074ac29b114b3bf0e3f693: Deleting consensus metadata
I20260927 22:09:31.379948 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:31.333635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42138 (local address 127.0.97.60:40003)
0927 22:09:31.333826 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:31.333834 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:31.363099 (+ 29265us) server_negotiation.cc:407] Connection header received
0927 22:09:31.363366 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:31.363372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:31.363452 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:31.363528 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:31.365314 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.366387 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.368417 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.369083 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.369185 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:31.372712 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:31.372719 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:31.372725 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:31.372764 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:31.376263 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:31.377104 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:31.377110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:31.377114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:31.377258 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:31.377793 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:31.377798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:31.377801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:31.378133 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:31.378382 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:31.379387 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0927 22:09:31.379613 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 22:09:31.383567 7859 ts_tablet_manager.cc:1918] T a294090772894adf95121fe922ddfc45 P ebdd20bf62754f138e1d99f4509ba2bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:09:31.383778 7859 log.cc:1196] T a294090772894adf95121fe922ddfc45 P ebdd20bf62754f138e1d99f4509ba2bd: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/a294090772894adf95121fe922ddfc45
I20260927 22:09:31.384425 7859 ts_tablet_manager.cc:1939] T a294090772894adf95121fe922ddfc45 P ebdd20bf62754f138e1d99f4509ba2bd: Deleting consensus metadata
I20260927 22:09:31.488517 6521 catalog_manager.cc:4841] TS 0f99af01ae61446fb3569c69c2e81936 (127.0.97.3:38105): tablet a294090772894adf95121fe922ddfc45 (table test-table [id=e9ab1e9591904b20a6df05d266e77aac]) successfully deleted
I20260927 22:09:31.489137 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 6505
W20260927 22:09:31.511471 7403 connection.cc:535] server connection from 127.0.0.1:51286 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 22:09:31.511530 7129 connection.cc:535] server connection from 127.0.0.1:48224 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 22:09:31.545909 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:31.516043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42150 (local address 127.0.97.60:40003)
0927 22:09:31.516248 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:31.516255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:31.517319 (+ 1064us) server_negotiation.cc:407] Connection header received
0927 22:09:31.517644 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:31.517653 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:31.517745 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:31.517826 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:31.531440 (+ 13614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.532557 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.533814 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.534447 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.534574 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:31.539177 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:31.539183 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:31.539188 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:31.539231 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:31.543094 (+ 3863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:31.544013 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:31.544017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:31.544020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:31.544166 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:31.544550 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:31.544554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:31.544558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:31.544811 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:31.545071 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:31.545411 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 22:09:31.545645 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:09:31.546669 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:31.517024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48282 (local address 127.0.97.61:39403)
0927 22:09:31.517214 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:31.517222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:31.528348 (+ 11126us) server_negotiation.cc:407] Connection header received
0927 22:09:31.528600 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:31.528607 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:31.528696 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:31.528797 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:31.532454 (+ 3657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.533669 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.535659 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:31.536064 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:31.536197 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:31.539803 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:31.539812 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:31.539817 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:31.539857 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:31.543192 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:31.543956 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:31.543964 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:31.543969 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:31.544114 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:31.545144 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:31.545149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:31.545152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:31.545484 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:31.545704 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:31.546088 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 22:09:31.546290 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:09:32.305480 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:32.304706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.97.3:34529)
0927 22:09:32.305166 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:32.305303 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":223,"threads_started":1}
W20260927 22:09:32.305841 7797 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35323 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
I20260927 22:09:32.312117 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:32.310325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.97.2:40179)
0927 22:09:32.311757 (+ 1432us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:32.311911 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1343,"thread_start_us":1249,"threads_started":1}
W20260927 22:09:32.312533 7656 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35323 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
I20260927 22:09:32.315594 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:32.314805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.97.1:37167)
0927 22:09:32.315254 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:32.315390 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":246,"threads_started":1}
W20260927 22:09:32.316370 7516 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35323 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
I20260927 22:09:32.829263 7873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae6791bd85ae4fceb7d3ac0a20c86d2d)
I20260927 22:09:32.829641 7873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: false } }
I20260927 22:09:32.832486 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:32.831343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.0.1:56014)
0927 22:09:32.831945 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:32.832147 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":253,"threads_started":1}
I20260927 22:09:32.833853 7873 leader_election.cc:288] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae6791bd85ae4fceb7d3ac0a20c86d2d (127.0.97.62:35323), ef79440c0b4c4fba8b6433b957ca03c3 (127.0.97.61:39403)
W20260927 22:09:32.834044 7321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
I20260927 22:09:32.835858 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:32.835319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.0.1:56020)
0927 22:09:32.835519 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:32.835656 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 22:09:32.836063 7321 leader_election.cc:334] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae6791bd85ae4fceb7d3ac0a20c86d2d (127.0.97.62:35323): Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
I20260927 22:09:32.839748 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:32.832128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39403 (local address 127.0.0.1:48292)
0927 22:09:32.832890 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:32.832911 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:32.833051 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:32.834890 (+ 1839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:32.834894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:32.834942 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:09:32.835161 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:32.835174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:32.836558 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:32.836566 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:32.837830 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:32.837841 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:32.838994 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:32.839000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:32.839175 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:32.839198 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:09:32.839313 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 22:09:32.839457 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":1543,"threads_started":1}
I20260927 22:09:32.839766 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:32.832457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48292 (local address 127.0.97.61:39403)
0927 22:09:32.834486 (+ 2029us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:32.834494 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:32.834516 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:09:32.834633 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:32.834640 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:32.834730 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:32.834847 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:09:32.835327 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:32.836421 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:32.837982 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:32.838835 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:32.838971 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:32.839073 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:32.839333 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 22:09:32.839478 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1878,"thread_start_us":1687,"threads_started":1}
I20260927 22:09:32.840531 7154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4cee22109f194af68fb1eb895cce38c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" is_pre_election: true
I20260927 22:09:32.841375 7321 leader_election.cc:302] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cee22109f194af68fb1eb895cce38c7; no voters: ae6791bd85ae4fceb7d3ac0a20c86d2d, ef79440c0b4c4fba8b6433b957ca03c3
I20260927 22:09:32.841747 7873 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 22:09:32.990794 7878 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae6791bd85ae4fceb7d3ac0a20c86d2d)
I20260927 22:09:32.991063 7878 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: false } }
I20260927 22:09:32.992403 7878 leader_election.cc:288] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae6791bd85ae4fceb7d3ac0a20c86d2d (127.0.97.62:35323), 4cee22109f194af68fb1eb895cce38c7 (127.0.97.60:40003)
I20260927 22:09:32.993520 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:32.992465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.0.1:56052)
0927 22:09:32.992964 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:32.993157 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":243,"threads_started":1}
W20260927 22:09:32.993862 7131 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
I20260927 22:09:33.004516 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.004022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.0.1:56060)
0927 22:09:33.004188 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:33.004300 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 22:09:33.005220 7131 leader_election.cc:334] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae6791bd85ae4fceb7d3ac0a20c86d2d (127.0.97.62:35323): Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
I20260927 22:09:33.009479 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:32.992667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40003 (local address 127.0.0.1:42154)
0927 22:09:32.999156 (+ 6489us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:32.999182 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:32.999348 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:33.004730 (+ 5382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:33.004733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:33.004755 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:09:33.004957 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:33.004965 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:33.006172 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:33.006180 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:33.007628 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:33.007639 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:33.008731 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:33.008737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:33.008896 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:33.008919 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:09:33.009044 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 22:09:33.009182 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6349,"thread_start_us":10184,"threads_started":1}
I20260927 22:09:33.009521 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:32.993168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42154 (local address 127.0.97.60:40003)
0927 22:09:33.004228 (+ 11060us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:33.004237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:33.004266 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:09:33.004349 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:33.004357 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:33.004492 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:33.004614 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:09:33.005135 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:33.006030 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:33.007772 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:33.008563 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:33.008706 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:33.008815 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:33.009065 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 22:09:33.009191 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10905,"thread_start_us":10757,"threads_started":1}
I20260927 22:09:33.010521 7343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4cee22109f194af68fb1eb895cce38c7" is_pre_election: true
I20260927 22:09:33.010897 7343 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef79440c0b4c4fba8b6433b957ca03c3 in term 1.
I20260927 22:09:33.011360 7130 leader_election.cc:302] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cee22109f194af68fb1eb895cce38c7, ef79440c0b4c4fba8b6433b957ca03c3; no voters: ae6791bd85ae4fceb7d3ac0a20c86d2d
I20260927 22:09:33.011708 7878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 22:09:33.011821 7878 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae6791bd85ae4fceb7d3ac0a20c86d2d)
I20260927 22:09:33.011932 7878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:09:33.014098 7878 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: false } }
I20260927 22:09:33.014709 7878 leader_election.cc:288] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [CANDIDATE]: Term 2 election: Requested vote from peers ae6791bd85ae4fceb7d3ac0a20c86d2d (127.0.97.62:35323), 4cee22109f194af68fb1eb895cce38c7 (127.0.97.60:40003)
I20260927 22:09:33.015103 7343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4cee22109f194af68fb1eb895cce38c7"
I20260927 22:09:33.015318 7343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:09:33.015399 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.014845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.0.1:56074)
0927 22:09:33.015104 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:33.015201 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260927 22:09:33.016448 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.016082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.0.1:56078)
0927 22:09:33.016230 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:33.016323 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 22:09:33.016680 7131 leader_election.cc:334] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ae6791bd85ae4fceb7d3ac0a20c86d2d (127.0.97.62:35323): Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
I20260927 22:09:33.017709 7343 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef79440c0b4c4fba8b6433b957ca03c3 in term 2.
I20260927 22:09:33.018087 7130 leader_election.cc:302] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cee22109f194af68fb1eb895cce38c7, ef79440c0b4c4fba8b6433b957ca03c3; no voters: ae6791bd85ae4fceb7d3ac0a20c86d2d
I20260927 22:09:33.018406 7878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:09:33.018934 7878 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [term 2 LEADER]: Becoming Leader. State: Replica: ef79440c0b4c4fba8b6433b957ca03c3, State: Running, Role: LEADER
I20260927 22:09:33.019461 7878 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: false } }
I20260927 22:09:33.021540 7883 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef79440c0b4c4fba8b6433b957ca03c3. Latest consensus state: current_term: 2 leader_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: false } } }
I20260927 22:09:33.021983 7883 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:33.022809 7885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:09:33.028064 7885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:09:33.029358 7885 catalog_manager.cc:1259] Loaded cluster ID: 0a21bad7669c4c3ea45c68ac34cabb13
I20260927 22:09:33.029459 7885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:09:33.031317 7885 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:09:33.032580 7885 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3: Loaded TSK: 0
I20260927 22:09:33.033896 7885 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:09:33.062147 7144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:09:33.069353 7144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:09:33.073519 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.073048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.0.1:56090)
0927 22:09:33.073258 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:33.073348 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 22:09:33.073526 7343 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [term 2 FOLLOWER]: Refusing update from remote peer ef79440c0b4c4fba8b6433b957ca03c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 22:09:33.074353 7883 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 22:09:33.074684 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.074270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.0.1:56106)
0927 22:09:33.074411 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:33.074526 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 22:09:33.074996 7131 consensus_peers.cc:487] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 -> Peer ae6791bd85ae4fceb7d3ac0a20c86d2d (127.0.97.62:35323): Couldn't send request to peer ae6791bd85ae4fceb7d3ac0a20c86d2d. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 22:09:33.079279 7873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef79440c0b4c4fba8b6433b957ca03c3. Latest consensus state: current_term: 2 leader_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: false } } }
I20260927 22:09:33.079591 7873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:09:33.080883 7886 mvcc.cc:205] Tried to move back new op lower bound from 7334080401703272448 to 7334080401493282816. Current Snapshot: MvccSnapshot[applied={T|T < 7334080401703272448}]
I20260927 22:09:33.081542 7878 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: false } } }
I20260927 22:09:33.081885 7878 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:33.082060 7873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ae6791bd85ae4fceb7d3ac0a20c86d2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } peers { permanent_uuid: "ef79440c0b4c4fba8b6433b957ca03c3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39403 } attrs { promote: false } } peers { permanent_uuid: "4cee22109f194af68fb1eb895cce38c7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40003 } attrs { promote: false } } }
I20260927 22:09:33.082322 7873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cee22109f194af68fb1eb895cce38c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:09:33.100014 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.090510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43179 (local address 127.0.0.1:51294)
0927 22:09:33.090700 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:33.090719 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:33.090838 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:33.093060 (+ 2222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:33.093068 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:33.093105 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:09:33.093579 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:33.093596 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:33.095407 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:33.095413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:33.097574 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:33.097593 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:33.099222 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:33.099228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:33.099422 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:33.099462 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 22:09:33.099533 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 22:09:33.099670 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 22:09:33.101089 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.091539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51294 (local address 127.0.97.1:43179)
0927 22:09:33.092054 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:33.092063 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:33.092090 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:09:33.092686 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:33.092697 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:33.092844 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:33.093811 (+ 967us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:09:33.094045 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:33.095255 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:33.097763 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:33.098977 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:33.100553 (+ 1576us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:33.100646 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:33.100694 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 22:09:33.100827 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":263,"threads_started":1}
I20260927 22:09:33.102115 7450 tablet_service.cc:1467] Processing CreateTablet for tablet 64b11fcc715c430dae703f86dd52e868 (DEFAULT_TABLE table=test-table [id=715027dd5ef148ca9d7975560f86d1aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:09:33.102613 7450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64b11fcc715c430dae703f86dd52e868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:33.103201 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.089925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38917 (local address 127.0.0.1:47952)
0927 22:09:33.090130 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:33.090152 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:33.090269 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:33.097398 (+ 7129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:33.097402 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:33.097426 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:09:33.097631 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:33.097639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:33.099068 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:33.099076 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:33.101037 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:33.101059 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:33.102421 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:33.102429 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:33.102648 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:33.102683 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:09:33.102751 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 22:09:33.102871 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":46}
I20260927 22:09:33.103739 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.091150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50154 (local address 127.0.97.3:38105)
0927 22:09:33.091718 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:33.091724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:33.095284 (+ 3560us) server_negotiation.cc:407] Connection header received
0927 22:09:33.095635 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:33.095644 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:33.095753 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:33.095836 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:09:33.096866 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:33.098074 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:33.101605 (+ 3531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:33.102536 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:33.102827 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:33.102957 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:33.103265 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 22:09:33.103404 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":300,"threads_started":1}
I20260927 22:09:33.103914 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.091130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38105 (local address 127.0.0.1:50154)
0927 22:09:33.095159 (+ 4029us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:33.095187 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:33.095399 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:33.096090 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:33.096097 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:33.096128 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:09:33.096666 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:33.096682 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:33.098246 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:33.098251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:33.101412 (+ 3161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:33.101429 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:33.102891 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:33.102898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:33.103090 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:33.103114 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 22:09:33.103239 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 22:09:33.103385 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3857,"mutex_wait_us":32,"thread_start_us":3654,"threads_started":1}
I20260927 22:09:33.103983 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.090005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47952 (local address 127.0.97.2:38917)
0927 22:09:33.097148 (+ 7143us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:33.097152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:33.097166 (+ 14us) server_negotiation.cc:407] Connection header received
0927 22:09:33.097219 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:33.097222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:33.097272 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:33.097385 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:09:33.097811 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:33.098897 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:33.101219 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:33.102240 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:33.103403 (+ 1163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:33.103491 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:33.103541 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 22:09:33.103668 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7038,"thread_start_us":6949,"threads_started":1}
I20260927 22:09:33.104624 7731 tablet_service.cc:1467] Processing CreateTablet for tablet 64b11fcc715c430dae703f86dd52e868 (DEFAULT_TABLE table=test-table [id=715027dd5ef148ca9d7975560f86d1aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:09:33.105123 7731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64b11fcc715c430dae703f86dd52e868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:33.107596 7590 tablet_service.cc:1467] Processing CreateTablet for tablet 64b11fcc715c430dae703f86dd52e868 (DEFAULT_TABLE table=test-table [id=715027dd5ef148ca9d7975560f86d1aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:09:33.108027 7590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64b11fcc715c430dae703f86dd52e868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:33.108835 7896 tablet_bootstrap.cc:492] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693: Bootstrap starting.
I20260927 22:09:33.110711 7896 tablet_bootstrap.cc:654] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:33.111657 7897 tablet_bootstrap.cc:492] T 64b11fcc715c430dae703f86dd52e868 P 0f99af01ae61446fb3569c69c2e81936: Bootstrap starting.
I20260927 22:09:33.113340 7897 tablet_bootstrap.cc:654] T 64b11fcc715c430dae703f86dd52e868 P 0f99af01ae61446fb3569c69c2e81936: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:33.113907 7896 tablet_bootstrap.cc:492] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693: No bootstrap required, opened a new log
I20260927 22:09:33.114130 7896 ts_tablet_manager.cc:1397] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 22:09:33.114280 7899 tablet_bootstrap.cc:492] T 64b11fcc715c430dae703f86dd52e868 P ebdd20bf62754f138e1d99f4509ba2bd: Bootstrap starting.
I20260927 22:09:33.115072 7896 raft_consensus.cc:348] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebdd20bf62754f138e1d99f4509ba2bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38917 } } peers { permanent_uuid: "100d5e2a45074ac29b114b3bf0e3f693" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43179 } } peers { permanent_uuid: "0f99af01ae61446fb3569c69c2e81936" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38105 } }
I20260927 22:09:33.115353 7896 raft_consensus.cc:374] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:33.115521 7896 raft_consensus.cc:729] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100d5e2a45074ac29b114b3bf0e3f693, State: Initialized, Role: FOLLOWER
I20260927 22:09:33.115804 7897 tablet_bootstrap.cc:492] T 64b11fcc715c430dae703f86dd52e868 P 0f99af01ae61446fb3569c69c2e81936: No bootstrap required, opened a new log
I20260927 22:09:33.115948 7897 ts_tablet_manager.cc:1397] T 64b11fcc715c430dae703f86dd52e868 P 0f99af01ae61446fb3569c69c2e81936: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 22:09:33.115851 7896 consensus_queue.cc:260] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebdd20bf62754f138e1d99f4509ba2bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38917 } } peers { permanent_uuid: "100d5e2a45074ac29b114b3bf0e3f693" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43179 } } peers { permanent_uuid: "0f99af01ae61446fb3569c69c2e81936" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38105 } }
I20260927 22:09:33.116396 7899 tablet_bootstrap.cc:654] T 64b11fcc715c430dae703f86dd52e868 P ebdd20bf62754f138e1d99f4509ba2bd: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:33.116717 7896 ts_tablet_manager.cc:1428] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:09:33.117022 7897 raft_consensus.cc:348] T 64b11fcc715c430dae703f86dd52e868 P 0f99af01ae61446fb3569c69c2e81936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebdd20bf62754f138e1d99f4509ba2bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38917 } } peers { permanent_uuid: "100d5e2a45074ac29b114b3bf0e3f693" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43179 } } peers { permanent_uuid: "0f99af01ae61446fb3569c69c2e81936" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38105 } }
I20260927 22:09:33.117369 7897 raft_consensus.cc:374] T 64b11fcc715c430dae703f86dd52e868 P 0f99af01ae61446fb3569c69c2e81936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:33.117550 7897 raft_consensus.cc:729] T 64b11fcc715c430dae703f86dd52e868 P 0f99af01ae61446fb3569c69c2e81936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f99af01ae61446fb3569c69c2e81936, State: Initialized, Role: FOLLOWER
I20260927 22:09:33.117885 7897 consensus_queue.cc:260] T 64b11fcc715c430dae703f86dd52e868 P 0f99af01ae61446fb3569c69c2e81936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebdd20bf62754f138e1d99f4509ba2bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38917 } } peers { permanent_uuid: "100d5e2a45074ac29b114b3bf0e3f693" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43179 } } peers { permanent_uuid: "0f99af01ae61446fb3569c69c2e81936" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38105 } }
I20260927 22:09:33.123145 7897 ts_tablet_manager.cc:1428] T 64b11fcc715c430dae703f86dd52e868 P 0f99af01ae61446fb3569c69c2e81936: Time spent starting tablet: real 0.007s user 0.003s sys 0.005s
I20260927 22:09:33.127018 7899 tablet_bootstrap.cc:492] T 64b11fcc715c430dae703f86dd52e868 P ebdd20bf62754f138e1d99f4509ba2bd: No bootstrap required, opened a new log
I20260927 22:09:33.127220 7899 ts_tablet_manager.cc:1397] T 64b11fcc715c430dae703f86dd52e868 P ebdd20bf62754f138e1d99f4509ba2bd: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260927 22:09:33.128088 7899 raft_consensus.cc:348] T 64b11fcc715c430dae703f86dd52e868 P ebdd20bf62754f138e1d99f4509ba2bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebdd20bf62754f138e1d99f4509ba2bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38917 } } peers { permanent_uuid: "100d5e2a45074ac29b114b3bf0e3f693" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43179 } } peers { permanent_uuid: "0f99af01ae61446fb3569c69c2e81936" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38105 } }
I20260927 22:09:33.128301 7899 raft_consensus.cc:374] T 64b11fcc715c430dae703f86dd52e868 P ebdd20bf62754f138e1d99f4509ba2bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:33.128412 7899 raft_consensus.cc:729] T 64b11fcc715c430dae703f86dd52e868 P ebdd20bf62754f138e1d99f4509ba2bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebdd20bf62754f138e1d99f4509ba2bd, State: Initialized, Role: FOLLOWER
I20260927 22:09:33.128676 7899 consensus_queue.cc:260] T 64b11fcc715c430dae703f86dd52e868 P ebdd20bf62754f138e1d99f4509ba2bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebdd20bf62754f138e1d99f4509ba2bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38917 } } peers { permanent_uuid: "100d5e2a45074ac29b114b3bf0e3f693" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43179 } } peers { permanent_uuid: "0f99af01ae61446fb3569c69c2e81936" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38105 } }
I20260927 22:09:33.129578 7899 ts_tablet_manager.cc:1428] T 64b11fcc715c430dae703f86dd52e868 P ebdd20bf62754f138e1d99f4509ba2bd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 22:09:33.132612 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.116968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.97.1:33959)
0927 22:09:33.132117 (+ 15149us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:33.132286 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14967,"threads_started":1}
I20260927 22:09:33.135949 7901 raft_consensus.cc:482] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:09:33.136328 7901 raft_consensus.cc:504] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebdd20bf62754f138e1d99f4509ba2bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38917 } } peers { permanent_uuid: "100d5e2a45074ac29b114b3bf0e3f693" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43179 } } peers { permanent_uuid: "0f99af01ae61446fb3569c69c2e81936" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38105 } }
I20260927 22:09:33.136659 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.124137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.97.3:58821)
0927 22:09:33.136289 (+ 12152us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:33.136421 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12006,"threads_started":1}
I20260927 22:09:33.137107 7901 leader_election.cc:288] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebdd20bf62754f138e1d99f4509ba2bd (127.0.97.2:38917), 0f99af01ae61446fb3569c69c2e81936 (127.0.97.3:38105)
I20260927 22:09:33.137530 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.131449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.97.2:33869)
0927 22:09:33.137134 (+ 5685us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:33.137271 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5557,"thread_start_us":5400,"threads_started":1}
I20260927 22:09:33.137562 7751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b11fcc715c430dae703f86dd52e868" candidate_uuid: "100d5e2a45074ac29b114b3bf0e3f693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f99af01ae61446fb3569c69c2e81936" is_pre_election: true
I20260927 22:09:33.137849 7751 raft_consensus.cc:2454] T 64b11fcc715c430dae703f86dd52e868 P 0f99af01ae61446fb3569c69c2e81936 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 100d5e2a45074ac29b114b3bf0e3f693 in term 0.
I20260927 22:09:33.138361 7403 leader_election.cc:302] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f99af01ae61446fb3569c69c2e81936, 100d5e2a45074ac29b114b3bf0e3f693; no voters:
I20260927 22:09:33.138675 7901 raft_consensus.cc:2781] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:09:33.138792 7901 raft_consensus.cc:482] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:09:33.138897 7901 raft_consensus.cc:3037] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:33.139312 7610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b11fcc715c430dae703f86dd52e868" candidate_uuid: "100d5e2a45074ac29b114b3bf0e3f693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebdd20bf62754f138e1d99f4509ba2bd" is_pre_election: true
I20260927 22:09:33.139564 7610 raft_consensus.cc:2454] T 64b11fcc715c430dae703f86dd52e868 P ebdd20bf62754f138e1d99f4509ba2bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 100d5e2a45074ac29b114b3bf0e3f693 in term 0.
I20260927 22:09:33.139794 7798 heartbeater.cc:503] Master 127.0.97.61:39403 was elected leader, sending a full tablet report...
I20260927 22:09:33.140075 7657 heartbeater.cc:503] Master 127.0.97.61:39403 was elected leader, sending a full tablet report...
I20260927 22:09:33.140149 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.139472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.97.3:42593)
0927 22:09:33.139641 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:33.139727 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 22:09:33.141762 7901 raft_consensus.cc:504] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebdd20bf62754f138e1d99f4509ba2bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38917 } } peers { permanent_uuid: "100d5e2a45074ac29b114b3bf0e3f693" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43179 } } peers { permanent_uuid: "0f99af01ae61446fb3569c69c2e81936" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38105 } }
I20260927 22:09:33.142179 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.138895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.97.2:56545)
0927 22:09:33.141965 (+ 3070us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:33.142055 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2966}
I20260927 22:09:33.142388 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.139351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.97.1:54925)
0927 22:09:33.142195 (+ 2844us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:33.142269 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2737}
I20260927 22:09:33.142814 7901 leader_election.cc:288] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693 [CANDIDATE]: Term 1 election: Requested vote from peers ebdd20bf62754f138e1d99f4509ba2bd (127.0.97.2:38917), 0f99af01ae61446fb3569c69c2e81936 (127.0.97.3:38105)
I20260927 22:09:33.142915 7751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b11fcc715c430dae703f86dd52e868" candidate_uuid: "100d5e2a45074ac29b114b3bf0e3f693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f99af01ae61446fb3569c69c2e81936"
I20260927 22:09:33.142993 7610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64b11fcc715c430dae703f86dd52e868" candidate_uuid: "100d5e2a45074ac29b114b3bf0e3f693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebdd20bf62754f138e1d99f4509ba2bd"
I20260927 22:09:33.143121 7751 raft_consensus.cc:3037] T 64b11fcc715c430dae703f86dd52e868 P 0f99af01ae61446fb3569c69c2e81936 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:33.143221 7610 raft_consensus.cc:3037] T 64b11fcc715c430dae703f86dd52e868 P ebdd20bf62754f138e1d99f4509ba2bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:33.145124 7751 raft_consensus.cc:2454] T 64b11fcc715c430dae703f86dd52e868 P 0f99af01ae61446fb3569c69c2e81936 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 100d5e2a45074ac29b114b3bf0e3f693 in term 1.
I20260927 22:09:33.145475 7610 raft_consensus.cc:2454] T 64b11fcc715c430dae703f86dd52e868 P ebdd20bf62754f138e1d99f4509ba2bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 100d5e2a45074ac29b114b3bf0e3f693 in term 1.
I20260927 22:09:33.145486 7403 leader_election.cc:302] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f99af01ae61446fb3569c69c2e81936, 100d5e2a45074ac29b114b3bf0e3f693; no voters:
I20260927 22:09:33.145750 7901 raft_consensus.cc:2781] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:09:33.146180 7901 raft_consensus.cc:686] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693 [term 1 LEADER]: Becoming Leader. State: Replica: 100d5e2a45074ac29b114b3bf0e3f693, State: Running, Role: LEADER
I20260927 22:09:33.146438 7901 consensus_queue.cc:237] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebdd20bf62754f138e1d99f4509ba2bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38917 } } peers { permanent_uuid: "100d5e2a45074ac29b114b3bf0e3f693" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43179 } } peers { permanent_uuid: "0f99af01ae61446fb3569c69c2e81936" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38105 } }
I20260927 22:09:33.146677 7517 heartbeater.cc:503] Master 127.0.97.61:39403 was elected leader, sending a full tablet report...
I20260927 22:09:33.147341 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.146794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.97.1:35991)
0927 22:09:33.147056 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:33.147160 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260927 22:09:33.149006 7144 catalog_manager.cc:5495] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693 reported cstate change: term changed from 0 to 1, leader changed from <none> to 100d5e2a45074ac29b114b3bf0e3f693 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "100d5e2a45074ac29b114b3bf0e3f693" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebdd20bf62754f138e1d99f4509ba2bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "100d5e2a45074ac29b114b3bf0e3f693" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f99af01ae61446fb3569c69c2e81936" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38105 } health_report { overall_health: UNKNOWN } } }
I20260927 22:09:33.164340 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.160286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51304 (local address 127.0.97.1:43179)
0927 22:09:33.160468 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:33.160473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:33.160488 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:09:33.160595 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:33.160598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:33.160658 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:33.160729 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:09:33.161083 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:33.161767 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:33.163047 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:33.163437 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:33.163523 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:33.163816 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:09:33.163909 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:09:33.163972 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:33.164087 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 22:09:33.164142 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 22:09:33.167166 7610 raft_consensus.cc:1261] T 64b11fcc715c430dae703f86dd52e868 P ebdd20bf62754f138e1d99f4509ba2bd [term 1 FOLLOWER]: Refusing update from remote peer 100d5e2a45074ac29b114b3bf0e3f693: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:09:33.167166 7751 raft_consensus.cc:1261] T 64b11fcc715c430dae703f86dd52e868 P 0f99af01ae61446fb3569c69c2e81936 [term 1 FOLLOWER]: Refusing update from remote peer 100d5e2a45074ac29b114b3bf0e3f693: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:09:33.167821 7908 consensus_queue.cc:1035] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebdd20bf62754f138e1d99f4509ba2bd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:09:33.168154 7901 consensus_queue.cc:1035] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f99af01ae61446fb3569c69c2e81936" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:09:33.171281 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.170746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.97.2:38175)
0927 22:09:33.170888 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:33.171027 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 22:09:33.171649 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.169925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.97.3:49569)
0927 22:09:33.171368 (+ 1443us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:33.171463 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1209}
I20260927 22:09:33.177939 7913 mvcc.cc:205] Tried to move back new op lower bound from 7334080402087723008 to 7334080402011246592. Current Snapshot: MvccSnapshot[applied={T|T < 7334080402087723008}]
I20260927 22:09:33.178985 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.178313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.97.3:59217)
0927 22:09:33.178620 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:33.178734 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260927 22:09:33.181291 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.180746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.97.2:56821)
0927 22:09:33.180950 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:33.181071 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 22:09:33.186477 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.185988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.97.1:53281)
0927 22:09:33.186198 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:33.186304 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 22:09:33.193185 7144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:09:33.193462 7144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:09:33.199458 7144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef79440c0b4c4fba8b6433b957ca03c3: Sending DeleteTablet for 3 replicas of tablet 64b11fcc715c430dae703f86dd52e868
I20260927 22:09:33.200441 7590 tablet_service.cc:1514] Processing DeleteTablet for tablet 64b11fcc715c430dae703f86dd52e868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:33 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:47952
I20260927 22:09:33.200536 7450 tablet_service.cc:1514] Processing DeleteTablet for tablet 64b11fcc715c430dae703f86dd52e868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:33 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:51294
I20260927 22:09:33.200742 7731 tablet_service.cc:1514] Processing DeleteTablet for tablet 64b11fcc715c430dae703f86dd52e868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:33 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:50154
I20260927 22:09:33.204288 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.203699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.97.1:44657)
0927 22:09:33.203901 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:33.204025 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 22:09:33.207082 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 7309
I20260927 22:09:33.211867 7922 tablet_replica.cc:330] T 64b11fcc715c430dae703f86dd52e868 P 0f99af01ae61446fb3569c69c2e81936: stopping tablet replica
I20260927 22:09:33.212142 7922 raft_consensus.cc:2229] T 64b11fcc715c430dae703f86dd52e868 P 0f99af01ae61446fb3569c69c2e81936 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:09:33.212486 7922 raft_consensus.cc:2258] T 64b11fcc715c430dae703f86dd52e868 P 0f99af01ae61446fb3569c69c2e81936 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:09:33.213394 7920 tablet_replica.cc:330] T 64b11fcc715c430dae703f86dd52e868 P ebdd20bf62754f138e1d99f4509ba2bd: stopping tablet replica
I20260927 22:09:33.213644 7920 raft_consensus.cc:2229] T 64b11fcc715c430dae703f86dd52e868 P ebdd20bf62754f138e1d99f4509ba2bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:09:33.213985 7920 raft_consensus.cc:2258] T 64b11fcc715c430dae703f86dd52e868 P ebdd20bf62754f138e1d99f4509ba2bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:09:33.215006 7922 ts_tablet_manager.cc:1905] T 64b11fcc715c430dae703f86dd52e868 P 0f99af01ae61446fb3569c69c2e81936: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:33.215047 7921 tablet_replica.cc:330] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693: stopping tablet replica
I20260927 22:09:33.215250 7921 raft_consensus.cc:2229] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:09:33.215615 7921 raft_consensus.cc:2258] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:09:33.216408 7921 ts_tablet_manager.cc:1905] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:33.219514 7920 ts_tablet_manager.cc:1905] T 64b11fcc715c430dae703f86dd52e868 P ebdd20bf62754f138e1d99f4509ba2bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:33.221647 7921 ts_tablet_manager.cc:1918] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:09:33.221781 7921 log.cc:1196] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-0/wal/wals/64b11fcc715c430dae703f86dd52e868
I20260927 22:09:33.222261 7921 ts_tablet_manager.cc:1939] T 64b11fcc715c430dae703f86dd52e868 P 100d5e2a45074ac29b114b3bf0e3f693: Deleting consensus metadata
I20260927 22:09:33.223269 7922 ts_tablet_manager.cc:1918] T 64b11fcc715c430dae703f86dd52e868 P 0f99af01ae61446fb3569c69c2e81936: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:09:33.223394 7922 log.cc:1196] T 64b11fcc715c430dae703f86dd52e868 P 0f99af01ae61446fb3569c69c2e81936: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/64b11fcc715c430dae703f86dd52e868
I20260927 22:09:33.223326 7131 catalog_manager.cc:4841] TS 100d5e2a45074ac29b114b3bf0e3f693 (127.0.97.1:43179): tablet 64b11fcc715c430dae703f86dd52e868 (table test-table [id=715027dd5ef148ca9d7975560f86d1aa]) successfully deleted
I20260927 22:09:33.223801 7922 ts_tablet_manager.cc:1939] T 64b11fcc715c430dae703f86dd52e868 P 0f99af01ae61446fb3569c69c2e81936: Deleting consensus metadata
I20260927 22:09:33.224856 7128 catalog_manager.cc:4841] TS 0f99af01ae61446fb3569c69c2e81936 (127.0.97.3:38105): tablet 64b11fcc715c430dae703f86dd52e868 (table test-table [id=715027dd5ef148ca9d7975560f86d1aa]) successfully deleted
I20260927 22:09:33.225737 7920 ts_tablet_manager.cc:1918] T 64b11fcc715c430dae703f86dd52e868 P ebdd20bf62754f138e1d99f4509ba2bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:09:33.225879 7920 log.cc:1196] T 64b11fcc715c430dae703f86dd52e868 P ebdd20bf62754f138e1d99f4509ba2bd: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/64b11fcc715c430dae703f86dd52e868
I20260927 22:09:33.226253 7920 ts_tablet_manager.cc:1939] T 64b11fcc715c430dae703f86dd52e868 P ebdd20bf62754f138e1d99f4509ba2bd: Deleting consensus metadata
I20260927 22:09:33.227177 7131 catalog_manager.cc:4841] TS ebdd20bf62754f138e1d99f4509ba2bd (127.0.97.2:38917): tablet 64b11fcc715c430dae703f86dd52e868 (table test-table [id=715027dd5ef148ca9d7975560f86d1aa]) successfully deleted
I20260927 22:09:33.232597 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.208451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51314 (local address 127.0.97.1:43179)
0927 22:09:33.208649 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:33.208658 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:33.218756 (+ 10098us) server_negotiation.cc:407] Connection header received
0927 22:09:33.219178 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:33.219186 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:33.219301 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:33.219391 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:33.221300 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:33.222397 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:33.224241 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:33.224829 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:33.224930 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:33.227686 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:33.227698 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:33.227715 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:33.227743 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:33.230396 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:33.231024 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:33.231031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:33.231033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:33.231151 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:33.231444 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:33.231448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:33.231449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:33.231758 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:33.231968 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:33.232144 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 22:09:33.232289 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:09:33.233211 7510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:33.359624 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 7309
I20260927 22:09:33.381156 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 7527
I20260927 22:09:33.393848 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.381825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47966 (local address 127.0.97.2:38917)
0927 22:09:33.382030 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:33.382038 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:33.382055 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:09:33.382204 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:33.382208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:33.382274 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:33.382358 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:33.383652 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:33.384425 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:33.385607 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:33.385996 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:33.386089 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:33.388714 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:33.388729 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:33.388750 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:33.388785 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:33.391477 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:33.392213 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:33.392219 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:33.392221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:33.392331 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:33.392666 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:33.392669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:33.392670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:33.393000 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:33.393199 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:33.393439 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 22:09:33.393599 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:09:33.394491 7650 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:33.478947 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.478493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.0.1:56112)
0927 22:09:33.478711 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:33.478809 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 22:09:33.479966 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.479656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.0.1:56118)
0927 22:09:33.479774 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:33.479841 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 22:09:33.486994 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 7527
I20260927 22:09:33.504705 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 7668
I20260927 22:09:33.517889 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:33.505504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50168 (local address 127.0.97.3:38105)
0927 22:09:33.505729 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:33.505735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:33.505755 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:09:33.505933 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:33.505944 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:33.506005 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:33.506108 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:33.507589 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:33.508428 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:33.509611 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:33.510024 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:33.510151 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:33.512581 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:33.512604 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:33.512620 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:33.512650 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:33.515339 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:33.515959 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:33.515970 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:33.515978 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:33.516089 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:33.516449 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:33.516456 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:33.516459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:33.516800 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:33.517079 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:33.517366 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 22:09:33.517586 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:09:33.518936 7791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:33.622165 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 7668
I20260927 22:09:33.638666 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 6999
I20260927 22:09:33.639778 7174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:33.760051 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 6999
I20260927 22:09:33.781281 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 7099
I20260927 22:09:33.782305 7364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:33.897027 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 7099
2026-09-27T22:09:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9710 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:09:34 dist-test-slave-mp03 krb5kdc[7931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:09:34 dist-test-slave-mp03 krb5kdc[7931](info): set up 2 sockets
Sep 27 22:09:34 dist-test-slave-mp03 krb5kdc[7931](info): commencing operation
krb5kdc: starting...
W20260927 22:09:34.995672 388 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:09:35 dist-test-slave-mp03 krb5kdc[7931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:09:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:09:35Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260927 22:09:35.230188 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42709
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:35.555614 7944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:35.555907 7944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:35.555998 7944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:35.563199 7944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:09:35.563298 7944 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 22:09:35.563339 7944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:35.563400 7944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:35.563449 7944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:09:35.563503 7944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:09:35.572956 7944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42709
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:35.574182 7944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:35.575549 7944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:35.587024 7949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:35.586969 7952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:35.587065 7950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:35.588358 7944 server_base.cc:1034] running on GCE node
I20260927 22:09:35.589077 7944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:35.590402 7944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:35.591615 7944 hybrid_clock.cc:648] HybridClock initialized: now 1790546975591541 us; error 47 us; skew 500 ppm
I20260927 22:09:35.591961 7944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:35 dist-test-slave-mp03 krb5kdc[7931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546975, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:35.595034 7944 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260927 22:09:35.596540 7944 webserver.cc:460] Webserver started at http://127.0.97.62:43195/ using document root <none> and password file <none>
I20260927 22:09:35.597219 7944 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:35.597350 7944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:35.597656 7944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:35.600507 7944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "f4f0ca4681ee418fb8f5d8926467ac7a"
format_stamp: "Formatted at 2026-09-27 22:09:35 on dist-test-slave-mp03"
server_key: "64a752a0d8c9b46e0a1855d489801c2a"
server_key_iv: "5e29be79b2eebde43c461f4e783da04c"
server_key_version: "encryptionkey@0"
I20260927 22:09:35.601290 7944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "f4f0ca4681ee418fb8f5d8926467ac7a"
format_stamp: "Formatted at 2026-09-27 22:09:35 on dist-test-slave-mp03"
server_key: "64a752a0d8c9b46e0a1855d489801c2a"
server_key_iv: "5e29be79b2eebde43c461f4e783da04c"
server_key_version: "encryptionkey@0"
I20260927 22:09:35.606192 7944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:09:35.610163 7959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:35.611814 7944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260927 22:09:35.612007 7944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "f4f0ca4681ee418fb8f5d8926467ac7a"
format_stamp: "Formatted at 2026-09-27 22:09:35 on dist-test-slave-mp03"
server_key: "64a752a0d8c9b46e0a1855d489801c2a"
server_key_iv: "5e29be79b2eebde43c461f4e783da04c"
server_key_version: "encryptionkey@0"
I20260927 22:09:35.612233 7944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:35.633529 7944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:35.638136 7944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:35.638404 7944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:35.660048 7944 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42709
I20260927 22:09:35.660055 8020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42709 every 8 connection(s)
I20260927 22:09:35.661459 7944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:09:35.665164 8021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:35.671401 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 7944
I20260927 22:09:35.671607 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:09:35.672462 388 external_mini_cluster.cc:1428] Setting key 4e8d788af2e39e4420327ffea3aa3600
I20260927 22:09:35.676481 8021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a: Bootstrap starting.
I20260927 22:09:35.679708 8021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:35.680754 8021 log.cc:824] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:35.684072 8021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a: No bootstrap required, opened a new log
Sep 27 22:09:35 dist-test-slave-mp03 krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:09:35.694839 8021 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f0ca4681ee418fb8f5d8926467ac7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42709 } }
I20260927 22:09:35.695323 8021 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:35.695447 8021 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f0ca4681ee418fb8f5d8926467ac7a, State: Initialized, Role: FOLLOWER
I20260927 22:09:35.696300 8021 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f0ca4681ee418fb8f5d8926467ac7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42709 } }
I20260927 22:09:35.696539 8021 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:09:35.696671 8021 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:09:35.696851 8021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:35.697672 8024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:35.674650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.0.97.62:42709)
0927 22:09:35.675993 (+ 1343us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:35.676006 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:35.676045 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:09:35.677803 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:35.677828 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:35.678231 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:35.678660 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:35.680171 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:35.681854 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:35.683713 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:35.684429 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:35.684566 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:35.689019 (+ 4453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:35.689030 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:35.689050 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:35.689100 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:35.693097 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:35.693835 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:35.693844 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:35.693853 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:35.694058 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:35.694409 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:35.694414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:35.694417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:35.695148 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:35.695476 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:35.695973 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 22:09:35.696438 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1108,"thread_start_us":886,"threads_started":1}
I20260927 22:09:35.699333 8021 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f0ca4681ee418fb8f5d8926467ac7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42709 } }
I20260927 22:09:35.699853 8021 leader_election.cc:302] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4f0ca4681ee418fb8f5d8926467ac7a; no voters:
I20260927 22:09:35.700488 8021 leader_election.cc:288] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:09:35.700621 8026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:09:35.701292 8026 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a [term 1 LEADER]: Becoming Leader. State: Replica: f4f0ca4681ee418fb8f5d8926467ac7a, State: Running, Role: LEADER
I20260927 22:09:35.701877 8026 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f0ca4681ee418fb8f5d8926467ac7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42709 } }
I20260927 22:09:35.702672 8021 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:09:35.704653 8028 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4f0ca4681ee418fb8f5d8926467ac7a. Latest consensus state: current_term: 1 leader_uuid: "f4f0ca4681ee418fb8f5d8926467ac7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f0ca4681ee418fb8f5d8926467ac7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42709 } } }
I20260927 22:09:35.705009 8028 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:35.705456 8027 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4f0ca4681ee418fb8f5d8926467ac7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f0ca4681ee418fb8f5d8926467ac7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42709 } } }
I20260927 22:09:35.705703 8027 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:35.712007 8033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:09:35.719599 8033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:09:35.728615 8033 catalog_manager.cc:1347] Generated new cluster ID: 703467767cf04ed7b078156c4afde965
I20260927 22:09:35.728746 8033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:09:35.765116 8033 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:09:35.766480 8033 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:09:35.776152 8033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a: Generated new TSK 0
I20260927 22:09:35.777331 8033 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260927 22:09:35.845345 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42709
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:36.174009 8047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:36.174342 8047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:36.174518 8047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:36.181823 8047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:36.182009 8047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:36.182188 8047 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:09:36.182339 8047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:09:36.192373 8047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:36.193771 8047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:36.195199 8047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:36.209224 8053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:36.209224 8055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:36.209329 8052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:36.209962 8047 server_base.cc:1034] running on GCE node
I20260927 22:09:36.210549 8047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:36.211534 8047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:36.212755 8047 hybrid_clock.cc:648] HybridClock initialized: now 1790546976212694 us; error 33 us; skew 500 ppm
I20260927 22:09:36.213106 8047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:36 dist-test-slave-mp03 krb5kdc[7931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546976, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:36.216492 8047 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 22:09:36.217978 8047 webserver.cc:460] Webserver started at http://127.0.97.1:32945/ using document root <none> and password file <none>
I20260927 22:09:36.218669 8047 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:36.218760 8047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:36.219029 8047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:36.221926 8047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "9055928a5514436eb0d1b85c292a7762"
format_stamp: "Formatted at 2026-09-27 22:09:36 on dist-test-slave-mp03"
server_key: "95900f11468d28ed6307d402e3102c02"
server_key_iv: "110908186ff438aa676afdaefdd76129"
server_key_version: "encryptionkey@0"
I20260927 22:09:36.222779 8047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9055928a5514436eb0d1b85c292a7762"
format_stamp: "Formatted at 2026-09-27 22:09:36 on dist-test-slave-mp03"
server_key: "95900f11468d28ed6307d402e3102c02"
server_key_iv: "110908186ff438aa676afdaefdd76129"
server_key_version: "encryptionkey@0"
I20260927 22:09:36.227752 8047 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:09:36.231176 8062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:36.232517 8047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:09:36.232688 8047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "9055928a5514436eb0d1b85c292a7762"
format_stamp: "Formatted at 2026-09-27 22:09:36 on dist-test-slave-mp03"
server_key: "95900f11468d28ed6307d402e3102c02"
server_key_iv: "110908186ff438aa676afdaefdd76129"
server_key_version: "encryptionkey@0"
I20260927 22:09:36.232901 8047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:36.245388 8047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:36.249753 8047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:36.250044 8047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:36.252769 8047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:36.252877 8047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:36.253006 8047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:36.253083 8047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:09:36 dist-test-slave-mp03 krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546976, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:09:36.279174 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:36.253397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50307 (local address 127.0.97.62:42709)
0927 22:09:36.254008 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:36.254013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:36.255081 (+ 1068us) server_negotiation.cc:407] Connection header received
0927 22:09:36.257198 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:36.257205 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:36.257306 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:36.257441 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:36.260016 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:36.261325 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:36.264084 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:36.264777 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:36.264938 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:36.270738 (+ 5800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:36.270751 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:36.270755 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:36.270790 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:36.274666 (+ 3876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:36.275912 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:36.275925 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:36.275931 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:36.276093 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:36.276818 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:36.276827 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:36.276830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:36.277249 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:36.277375 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:36.278487 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0927 22:09:36.278752 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":411,"threads_started":1}
I20260927 22:09:36.281067 8072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:36.254044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42709 (local address 127.0.97.1:50307)
0927 22:09:36.254917 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:36.254961 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:36.256917 (+ 1956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:36.257716 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:36.257740 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:36.258703 (+ 963us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:36.259733 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:36.259763 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:36.261518 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:36.261531 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:36.263920 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:36.263933 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:36.265058 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:36.265072 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:36.265456 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:36.266321 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:36.266357 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:36.270404 (+ 4047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:36.274848 (+ 4444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:36.274865 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:36.274892 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:36.275707 (+ 815us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:36.276280 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:36.276298 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:36.276302 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:36.276629 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:36.277405 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:36.277417 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:36.277992 (+ 575us) client_negotiation.cc:769] Sending connection context
0927 22:09:36.278383 (+ 391us) client_negotiation.cc:240] Negotiation successful
0927 22:09:36.279118 (+ 735us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":349,"threads_started":1}
I20260927 22:09:36.309892 8047 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38497
I20260927 22:09:36.309911 8182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38497 every 8 connection(s)
I20260927 22:09:36.311088 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:36.294765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52729 (local address 127.0.97.62:42709)
0927 22:09:36.295008 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:36.295020 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:36.295684 (+ 664us) server_negotiation.cc:407] Connection header received
0927 22:09:36.295952 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:36.295960 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:36.296058 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:36.296171 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:36.297697 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:36.298495 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:36.301092 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:36.301510 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:36.301699 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:36.304599 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:36.304607 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:36.304615 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:36.304660 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:36.308158 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:36.309112 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:36.309118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:36.309121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:36.309250 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:36.309648 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:36.309653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:36.309656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:36.309917 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:36.310080 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:36.310463 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 22:09:36.310704 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260927 22:09:36.311175 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:36.294837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42709 (local address 127.0.97.1:52729)
0927 22:09:36.295554 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:36.295570 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:36.295766 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:36.296230 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:36.296238 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:36.297005 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:36.297515 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:36.297532 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:36.298717 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:36.298726 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:36.300951 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:36.300968 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:36.301809 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:36.301816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:36.302118 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:36.303078 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:36.303109 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:36.304395 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:36.308298 (+ 3903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:36.308306 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:36.308313 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:36.308975 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:36.309381 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:36.309386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:36.309389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:36.309528 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:36.310039 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:36.310044 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:36.310223 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 22:09:36.310365 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 22:09:36.310721 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":420,"threads_started":1}
I20260927 22:09:36.311432 8047 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:09:36.313611 8183 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42709
I20260927 22:09:36.313838 8183 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:36.314743 8183 heartbeater.cc:511] Master 127.0.97.62:42709 requested a full tablet report, sending...
I20260927 22:09:36.316730 7976 ts_manager.cc:194] Registered new tserver with Master: 9055928a5514436eb0d1b85c292a7762 (127.0.97.1:38497)
I20260927 22:09:36.318146 7976 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50307
I20260927 22:09:36.318843 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 8047
I20260927 22:09:36.319047 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:09:36.319726 388 external_mini_cluster.cc:1428] Setting key bfba253b6ca702c7492dfe28c93a0628
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260927 22:09:36.368189 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42709
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:36.700522 8187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:36.700776 8187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:36.700870 8187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:36.707724 8187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:36.707825 8187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:36.707906 8187 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:09:36.707985 8187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:09:36.717747 8187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:36.719062 8187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:36.720516 8187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:36.733382 8192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:36.733304 8193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:36.733358 8195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:36.733878 8187 server_base.cc:1034] running on GCE node
I20260927 22:09:36.734539 8187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:36.735430 8187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:36.736675 8187 hybrid_clock.cc:648] HybridClock initialized: now 1790546976736561 us; error 54 us; skew 500 ppm
I20260927 22:09:36.736971 8187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:36 dist-test-slave-mp03 krb5kdc[7931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546976, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:36.740028 8187 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 22:09:36.741345 8187 webserver.cc:460] Webserver started at http://127.0.97.2:45047/ using document root <none> and password file <none>
I20260927 22:09:36.741999 8187 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:36.742106 8187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:36.742429 8187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:36.745215 8187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "7f91d96917494a90aa3515a177f867e3"
format_stamp: "Formatted at 2026-09-27 22:09:36 on dist-test-slave-mp03"
server_key: "d51bdb7951d8ebe3402ceb5664280132"
server_key_iv: "8d79d9349ea37c9a7207d4540fdd813f"
server_key_version: "encryptionkey@0"
I20260927 22:09:36.745958 8187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7f91d96917494a90aa3515a177f867e3"
format_stamp: "Formatted at 2026-09-27 22:09:36 on dist-test-slave-mp03"
server_key: "d51bdb7951d8ebe3402ceb5664280132"
server_key_iv: "8d79d9349ea37c9a7207d4540fdd813f"
server_key_version: "encryptionkey@0"
I20260927 22:09:36.750850 8187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 22:09:36.754145 8202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:36.755370 8187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:09:36.755542 8187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "7f91d96917494a90aa3515a177f867e3"
format_stamp: "Formatted at 2026-09-27 22:09:36 on dist-test-slave-mp03"
server_key: "d51bdb7951d8ebe3402ceb5664280132"
server_key_iv: "8d79d9349ea37c9a7207d4540fdd813f"
server_key_version: "encryptionkey@0"
I20260927 22:09:36.755766 8187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:36.776026 8187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:36.779945 8187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:36.780221 8187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:36.782809 8187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:36.782909 8187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:36.783035 8187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:36.783130 8187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:09:36 dist-test-slave-mp03 krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546976, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:09:36.808904 8211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:36.784060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42709 (local address 127.0.97.2:59271)
0927 22:09:36.784830 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:36.784871 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:36.786761 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:36.787572 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:36.787588 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:36.788430 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:36.789227 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:36.789248 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:36.790569 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:36.790578 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:36.793348 (+ 2770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:36.793370 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:36.794190 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:36.794201 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:36.794581 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:36.795413 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:36.795456 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:36.799670 (+ 4214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:36.803605 (+ 3935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:36.803618 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:36.803635 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:36.804302 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:36.804749 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:36.804754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:36.804758 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:36.805078 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:36.805814 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:36.805820 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:36.806370 (+ 550us) client_negotiation.cc:769] Sending connection context
0927 22:09:36.806728 (+ 358us) client_negotiation.cc:240] Negotiation successful
0927 22:09:36.807376 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":315,"threads_started":1}
I20260927 22:09:36.809319 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:36.783422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59271 (local address 127.0.97.62:42709)
0927 22:09:36.783665 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:36.783672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:36.784976 (+ 1304us) server_negotiation.cc:407] Connection header received
0927 22:09:36.787062 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:36.787067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:36.787159 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:36.787273 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:36.789394 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:36.790385 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:36.793508 (+ 3123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:36.793912 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:36.794012 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:36.799977 (+ 5965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:36.799985 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:36.799991 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:36.800042 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:36.803461 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:36.804480 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:36.804487 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:36.804493 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:36.804639 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:36.805197 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:36.805202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:36.805205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:36.805593 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:36.805752 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:36.808798 (+ 3046us) server_negotiation.cc:299] Negotiation successful
0927 22:09:36.808953 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 22:09:36.827016 8187 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45337
I20260927 22:09:36.827021 8321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45337 every 8 connection(s)
I20260927 22:09:36.828644 8187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:09:36.829257 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 8187
I20260927 22:09:36.829420 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:09:36.830006 388 external_mini_cluster.cc:1428] Setting key ff31f1537bf2c1c96a06c17c4e022b18
I20260927 22:09:36.831110 8322 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42709
I20260927 22:09:36.831413 8322 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:36.832720 8297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:36.816333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42709 (local address 127.0.97.2:55443)
0927 22:09:36.817024 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:36.817042 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:36.817210 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:36.817753 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:36.817756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:36.818443 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:36.819037 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:36.819056 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:36.820502 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:36.820509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:36.821576 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:36.821587 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:36.822792 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:36.822799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:36.823032 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:36.823793 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:36.823823 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:36.824931 (+ 1108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:36.828911 (+ 3980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:36.828915 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:36.828919 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:36.829512 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:36.829943 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:36.829948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:36.829951 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:36.830101 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:36.831138 (+ 1037us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:36.831146 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:36.831423 (+ 277us) client_negotiation.cc:769] Sending connection context
0927 22:09:36.831604 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 22:09:36.832103 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":415,"threads_started":1}
I20260927 22:09:36.832370 8322 heartbeater.cc:511] Master 127.0.97.62:42709 requested a full tablet report, sending...
I20260927 22:09:36.833313 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:36.816252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55443 (local address 127.0.97.62:42709)
0927 22:09:36.816422 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:36.816428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:36.817135 (+ 707us) server_negotiation.cc:407] Connection header received
0927 22:09:36.817373 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:36.817381 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:36.817473 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:36.817549 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:36.819180 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:36.820365 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:36.821713 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:36.822348 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:36.822472 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:36.825144 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:36.825150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:36.825154 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:36.825191 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:36.828758 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:36.829667 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:36.829673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:36.829677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:36.829815 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:36.830238 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:36.830246 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:36.830253 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:36.830849 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:36.831042 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:36.832546 (+ 1504us) server_negotiation.cc:299] Negotiation successful
0927 22:09:36.832843 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 22:09:36.834841 7976 ts_manager.cc:194] Registered new tserver with Master: 7f91d96917494a90aa3515a177f867e3 (127.0.97.2:45337)
I20260927 22:09:36.836292 7976 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59271
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260927 22:09:36.878259 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42709
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:37.197726 8326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:37.198071 8326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:37.198243 8326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:37.205682 8326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:37.205826 8326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:37.205976 8326 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:09:37.206118 8326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:09:37.216151 8326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:37.217603 8326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:37.219030 8326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:37.232048 8331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:37.232270 8332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:37.232539 8334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:37.232926 8326 server_base.cc:1034] running on GCE node
I20260927 22:09:37.233512 8326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:37.234416 8326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:37.235664 8326 hybrid_clock.cc:648] HybridClock initialized: now 1790546977235588 us; error 59 us; skew 500 ppm
I20260927 22:09:37.235973 8326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:37 dist-test-slave-mp03 krb5kdc[7931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:37.239245 8326 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 22:09:37.240669 8326 webserver.cc:460] Webserver started at http://127.0.97.3:46685/ using document root <none> and password file <none>
I20260927 22:09:37.241384 8326 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:37.241510 8326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:37.241812 8326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:37.244657 8326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "06e458c806684e1fbe62ed518c3cbc23"
format_stamp: "Formatted at 2026-09-27 22:09:37 on dist-test-slave-mp03"
server_key: "e4cd6d76356b870450feeac53d5c4804"
server_key_iv: "c4f6fd680907168a23c4f5239b5b93ee"
server_key_version: "encryptionkey@0"
I20260927 22:09:37.245420 8326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "06e458c806684e1fbe62ed518c3cbc23"
format_stamp: "Formatted at 2026-09-27 22:09:37 on dist-test-slave-mp03"
server_key: "e4cd6d76356b870450feeac53d5c4804"
server_key_iv: "c4f6fd680907168a23c4f5239b5b93ee"
server_key_version: "encryptionkey@0"
I20260927 22:09:37.250381 8326 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 22:09:37.253787 8341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:37.255115 8326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:09:37.255277 8326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "06e458c806684e1fbe62ed518c3cbc23"
format_stamp: "Formatted at 2026-09-27 22:09:37 on dist-test-slave-mp03"
server_key: "e4cd6d76356b870450feeac53d5c4804"
server_key_iv: "c4f6fd680907168a23c4f5239b5b93ee"
server_key_version: "encryptionkey@0"
I20260927 22:09:37.255489 8326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:37.276839 8326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:37.280766 8326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:37.281042 8326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:37.283833 8326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:37.283952 8326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:37.284104 8326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:37.284183 8326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:09:37 dist-test-slave-mp03 krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:09:37.307842 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:37.284346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35917 (local address 127.0.97.62:42709)
0927 22:09:37.284579 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:37.284588 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:37.285974 (+ 1386us) server_negotiation.cc:407] Connection header received
0927 22:09:37.287988 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:37.287993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:37.288088 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:37.288197 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:37.290294 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.291643 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.294221 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.295011 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.295156 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:37.300433 (+ 5277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:37.300438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:37.300441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:37.300482 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:37.303650 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:37.304809 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:37.304813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:37.304815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:37.304942 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:37.305541 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:37.305544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:37.305546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:37.305802 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:37.305898 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:37.307105 (+ 1207us) server_negotiation.cc:299] Negotiation successful
0927 22:09:37.307373 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 22:09:37.310253 8350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:37.285017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42709 (local address 127.0.97.3:35917)
0927 22:09:37.285806 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:37.285845 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:37.287702 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:37.288436 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:37.288444 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:37.289214 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:37.290056 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:37.290087 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.291855 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.291862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:37.294090 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:37.294103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.295327 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.295334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:37.295536 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:37.296477 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:37.296522 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:37.300098 (+ 3576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:37.303809 (+ 3711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:37.303823 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:37.303849 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:37.304671 (+ 822us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:37.305072 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:37.305080 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:37.305087 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:37.305428 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:37.305939 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:37.305951 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:37.306547 (+ 596us) client_negotiation.cc:769] Sending connection context
0927 22:09:37.306998 (+ 451us) client_negotiation.cc:240] Negotiation successful
0927 22:09:37.308253 (+ 1255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":353,"threads_started":1}
I20260927 22:09:37.321094 8183 heartbeater.cc:503] Master 127.0.97.62:42709 was elected leader, sending a full tablet report...
I20260927 22:09:37.335273 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:37.317793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42709 (local address 127.0.97.3:52089)
0927 22:09:37.318659 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:37.318676 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:37.318843 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:37.319163 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:37.319167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:37.319675 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:37.320028 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:37.320038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.321070 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.321078 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:37.323269 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:37.323289 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.324106 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.324117 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:37.324376 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:37.325166 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:37.325206 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:37.326705 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:37.331797 (+ 5092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:37.331804 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:37.331810 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:37.332324 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:37.332802 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:37.332806 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:37.332808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:37.332953 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:37.333726 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:37.333730 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:37.333948 (+ 218us) client_negotiation.cc:769] Sending connection context
0927 22:09:37.334146 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 22:09:37.334534 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":459,"threads_started":1}
I20260927 22:09:37.335335 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:37.317366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52089 (local address 127.0.97.62:42709)
0927 22:09:37.317579 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:37.317586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:37.318765 (+ 1179us) server_negotiation.cc:407] Connection header received
0927 22:09:37.318965 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:37.318970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:37.319044 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:37.319141 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:37.320164 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.320923 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.323432 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.323902 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.324053 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:37.326967 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:37.326977 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:37.326985 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:37.327034 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:37.331640 (+ 4606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:37.332501 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:37.332509 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:37.332514 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:37.332685 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:37.333109 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:37.333117 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:37.333121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:37.333517 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:37.333664 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:37.334593 (+ 929us) server_negotiation.cc:299] Negotiation successful
0927 22:09:37.334901 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 22:09:37.344763 8326 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39595
I20260927 22:09:37.344786 8460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39595 every 8 connection(s)
I20260927 22:09:37.346036 8326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:09:37.348162 8461 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42709
I20260927 22:09:37.348380 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 8326
I20260927 22:09:37.348400 8461 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:37.348556 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:09:37.348970 388 external_mini_cluster.cc:1428] Setting key cee7475c1f41ad2e7ad4c0ef1776622e
I20260927 22:09:37.349112 8461 heartbeater.cc:511] Master 127.0.97.62:42709 requested a full tablet report, sending...
I20260927 22:09:37.350617 7976 ts_manager.cc:194] Registered new tserver with Master: 06e458c806684e1fbe62ed518c3cbc23 (127.0.97.3:39595)
I20260927 22:09:37.351568 7976 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35917
I20260927 22:09:37.351850 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:09:37 dist-test-slave-mp03 krb5kdc[7931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546977, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:09:37 dist-test-slave-mp03 krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:37 dist-test-slave-mp03 krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:09:37.402186 8071 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:37.390768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56584 (local address 127.0.97.62:42709)
0927 22:09:37.391033 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:37.391041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:37.391288 (+ 247us) server_negotiation.cc:407] Connection header received
0927 22:09:37.391520 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:37.391526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:37.391609 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:37.391748 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:37.393263 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.394174 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.395586 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.396069 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.396226 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:37.401883 (+ 5657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56584: BlockingRecv error: recv got EOF from 127.0.0.1:56584 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 27 22:09:37 dist-test-slave-mp03 krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546977, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:09:37.418713 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:37.405394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56600 (local address 127.0.97.62:42709)
0927 22:09:37.405551 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:37.405556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:37.405966 (+ 410us) server_negotiation.cc:407] Connection header received
0927 22:09:37.406097 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:37.406101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:37.406173 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:37.406255 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:37.407418 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.408133 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.409300 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.409694 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.409782 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:37.413795 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:37.413800 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:37.413803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:37.413833 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:37.416513 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:37.417209 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:37.417212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:37.417215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:37.417320 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:37.417630 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:37.417632 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:37.417634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:37.417875 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:37.418073 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:37.418264 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 22:09:37.418427 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:09:37.422583 7976 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:09:37.429219 7976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:09:37.451056 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:37.441217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38968 (local address 127.0.97.2:45337)
0927 22:09:37.441778 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:37.441811 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:37.442242 (+ 431us) server_negotiation.cc:407] Connection header received
0927 22:09:37.442558 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:37.442570 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:37.442906 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:37.443094 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:09:37.444011 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.445464 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.448158 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.449585 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.449708 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:37.449917 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:37.450526 (+ 609us) server_negotiation.cc:299] Negotiation successful
0927 22:09:37.450748 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":250,"threads_started":1}
I20260927 22:09:37.451124 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:37.440802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45337 (local address 127.0.0.1:38968)
0927 22:09:37.442104 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:37.442133 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:37.442336 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:37.443334 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:37.443341 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:37.443368 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:09:37.443813 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:37.443826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.445606 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.445613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:37.447952 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:37.447963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.450038 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.450045 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:37.450259 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:37.450318 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 22:09:37.450451 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 22:09:37.450675 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1127,"mutex_wait_us":25,"thread_start_us":1621,"threads_started":1}
I20260927 22:09:37.453469 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:37.441190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39595 (local address 127.0.0.1:50978)
0927 22:09:37.442942 (+ 1752us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:37.442961 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:37.443078 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:37.444820 (+ 1742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:37.444824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:37.444849 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:09:37.445238 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:37.445247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.448554 (+ 3307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.448559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:37.450682 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:37.450694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.452702 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.452708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:37.452875 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:37.452906 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:09:37.452994 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 22:09:37.453116 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1608,"mutex_wait_us":5,"thread_start_us":2281,"threads_started":1}
I20260927 22:09:37.453516 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:37.441907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50978 (local address 127.0.97.3:39595)
0927 22:09:37.442539 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:37.442547 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:37.443220 (+ 673us) server_negotiation.cc:407] Connection header received
0927 22:09:37.443322 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:37.443330 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:37.443570 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:37.443760 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:09:37.445656 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.448402 (+ 2746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.450855 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.452516 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.452651 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:37.452857 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:37.453049 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 22:09:37.453202 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":335,"threads_started":1}
I20260927 22:09:37.455264 8252 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb86a7786e44fbaba65c09c56a485b2 (DEFAULT_TABLE table=test-table [id=1e7e5758a56c4559a0d9ed0e101a26bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:09:37.456771 8252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb86a7786e44fbaba65c09c56a485b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:37.457867 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:37.443190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38497 (local address 127.0.0.1:35776)
0927 22:09:37.449882 (+ 6692us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:37.449903 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:37.450022 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:37.450909 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:37.450913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:37.450935 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:09:37.451932 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:37.451944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.453745 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.453750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:37.455419 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:37.455430 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.457170 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.457174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:37.457289 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:37.457309 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 22:09:37.457417 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 22:09:37.457541 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6544,"thread_start_us":6387,"threads_started":1}
I20260927 22:09:37.457393 8391 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb86a7786e44fbaba65c09c56a485b2 (DEFAULT_TABLE table=test-table [id=1e7e5758a56c4559a0d9ed0e101a26bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:09:37.457901 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:37.443654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35776 (local address 127.0.97.1:38497)
0927 22:09:37.450099 (+ 6445us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:37.450110 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:37.450140 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:09:37.450228 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:37.450235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:37.450503 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:37.450715 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:09:37.452119 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.453588 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.455567 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.456763 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.456889 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:37.457071 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:37.457467 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 22:09:37.457613 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6284,"thread_start_us":6429,"threads_started":1}
I20260927 22:09:37.458832 8391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb86a7786e44fbaba65c09c56a485b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:37.461627 8113 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb86a7786e44fbaba65c09c56a485b2 (DEFAULT_TABLE table=test-table [id=1e7e5758a56c4559a0d9ed0e101a26bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:09:37.462697 8113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb86a7786e44fbaba65c09c56a485b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:37.473047 8486 tablet_bootstrap.cc:492] T 9cb86a7786e44fbaba65c09c56a485b2 P 9055928a5514436eb0d1b85c292a7762: Bootstrap starting.
I20260927 22:09:37.474390 8485 tablet_bootstrap.cc:492] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3: Bootstrap starting.
I20260927 22:09:37.477402 8486 tablet_bootstrap.cc:654] T 9cb86a7786e44fbaba65c09c56a485b2 P 9055928a5514436eb0d1b85c292a7762: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:37.477655 8485 tablet_bootstrap.cc:654] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:37.478730 8485 log.cc:824] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:37.478730 8486 log.cc:824] T 9cb86a7786e44fbaba65c09c56a485b2 P 9055928a5514436eb0d1b85c292a7762: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:37.479578 8487 tablet_bootstrap.cc:492] T 9cb86a7786e44fbaba65c09c56a485b2 P 06e458c806684e1fbe62ed518c3cbc23: Bootstrap starting.
I20260927 22:09:37.482439 8487 tablet_bootstrap.cc:654] T 9cb86a7786e44fbaba65c09c56a485b2 P 06e458c806684e1fbe62ed518c3cbc23: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:37.483743 8487 log.cc:824] T 9cb86a7786e44fbaba65c09c56a485b2 P 06e458c806684e1fbe62ed518c3cbc23: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:37.485368 8486 tablet_bootstrap.cc:492] T 9cb86a7786e44fbaba65c09c56a485b2 P 9055928a5514436eb0d1b85c292a7762: No bootstrap required, opened a new log
I20260927 22:09:37.485646 8486 ts_tablet_manager.cc:1397] T 9cb86a7786e44fbaba65c09c56a485b2 P 9055928a5514436eb0d1b85c292a7762: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260927 22:09:37.485942 8485 tablet_bootstrap.cc:492] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3: No bootstrap required, opened a new log
I20260927 22:09:37.486191 8485 ts_tablet_manager.cc:1397] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260927 22:09:37.486297 8487 tablet_bootstrap.cc:492] T 9cb86a7786e44fbaba65c09c56a485b2 P 06e458c806684e1fbe62ed518c3cbc23: No bootstrap required, opened a new log
I20260927 22:09:37.486514 8487 ts_tablet_manager.cc:1397] T 9cb86a7786e44fbaba65c09c56a485b2 P 06e458c806684e1fbe62ed518c3cbc23: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 22:09:37.493506 8487 raft_consensus.cc:348] T 9cb86a7786e44fbaba65c09c56a485b2 P 06e458c806684e1fbe62ed518c3cbc23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } } peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } }
I20260927 22:09:37.493870 8487 raft_consensus.cc:374] T 9cb86a7786e44fbaba65c09c56a485b2 P 06e458c806684e1fbe62ed518c3cbc23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:37.493963 8487 raft_consensus.cc:729] T 9cb86a7786e44fbaba65c09c56a485b2 P 06e458c806684e1fbe62ed518c3cbc23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e458c806684e1fbe62ed518c3cbc23, State: Initialized, Role: FOLLOWER
I20260927 22:09:37.494556 8487 consensus_queue.cc:260] T 9cb86a7786e44fbaba65c09c56a485b2 P 06e458c806684e1fbe62ed518c3cbc23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } } peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } }
I20260927 22:09:37.495646 8461 heartbeater.cc:503] Master 127.0.97.62:42709 was elected leader, sending a full tablet report...
I20260927 22:09:37.496344 8487 ts_tablet_manager.cc:1428] T 9cb86a7786e44fbaba65c09c56a485b2 P 06e458c806684e1fbe62ed518c3cbc23: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260927 22:09:37.496547 8486 raft_consensus.cc:348] T 9cb86a7786e44fbaba65c09c56a485b2 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } } peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } }
I20260927 22:09:37.496548 8485 raft_consensus.cc:348] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } } peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } }
I20260927 22:09:37.496948 8485 raft_consensus.cc:374] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:37.496953 8486 raft_consensus.cc:374] T 9cb86a7786e44fbaba65c09c56a485b2 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:37.497051 8486 raft_consensus.cc:729] T 9cb86a7786e44fbaba65c09c56a485b2 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9055928a5514436eb0d1b85c292a7762, State: Initialized, Role: FOLLOWER
I20260927 22:09:37.497049 8485 raft_consensus.cc:729] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f91d96917494a90aa3515a177f867e3, State: Initialized, Role: FOLLOWER
I20260927 22:09:37.497614 8485 consensus_queue.cc:260] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } } peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } }
I20260927 22:09:37.498548 8322 heartbeater.cc:503] Master 127.0.97.62:42709 was elected leader, sending a full tablet report...
I20260927 22:09:37.499274 8485 ts_tablet_manager.cc:1428] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260927 22:09:37.499228 8486 consensus_queue.cc:260] T 9cb86a7786e44fbaba65c09c56a485b2 P 9055928a5514436eb0d1b85c292a7762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } } peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } }
I20260927 22:09:37.501097 8486 ts_tablet_manager.cc:1428] T 9cb86a7786e44fbaba65c09c56a485b2 P 9055928a5514436eb0d1b85c292a7762: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
W20260927 22:09:37.564173 8184 tablet.cc:2369] T 9cb86a7786e44fbaba65c09c56a485b2 P 9055928a5514436eb0d1b85c292a7762: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:09:37.592507 8323 tablet.cc:2369] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:09:37.597609 8462 tablet.cc:2369] T 9cb86a7786e44fbaba65c09c56a485b2 P 06e458c806684e1fbe62ed518c3cbc23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:09:37.769147 8492 raft_consensus.cc:482] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:09:37.769543 8492 raft_consensus.cc:504] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } } peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } }
I20260927 22:09:37.771294 8492 leader_election.cc:288] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06e458c806684e1fbe62ed518c3cbc23 (127.0.97.3:39595), 9055928a5514436eb0d1b85c292a7762 (127.0.97.1:38497)
I20260927 22:09:37.778187 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:37.771767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40213 (local address 127.0.97.1:38497)
0927 22:09:37.771976 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:37.771984 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:37.772788 (+ 804us) server_negotiation.cc:407] Connection header received
0927 22:09:37.772994 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:37.773005 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:37.773073 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:37.773171 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:09:37.773784 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.774477 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.776512 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.777276 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.777432 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:37.777533 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:37.777814 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 22:09:37.777933 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:09:37.778301 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:37.771691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46247 (local address 127.0.97.3:39595)
0927 22:09:37.771909 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:37.771918 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:37.772872 (+ 954us) server_negotiation.cc:407] Connection header received
0927 22:09:37.773056 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:37.773061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:37.773129 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:37.773223 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:09:37.773794 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.774469 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.776644 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.777398 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.777552 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:37.777671 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:37.777934 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 22:09:37.778055 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 22:09:37.778347 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:37.771663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38497 (local address 127.0.97.2:40213)
0927 22:09:37.772669 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:37.772684 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:37.772861 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:37.773188 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:37.773192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:37.773215 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:09:37.773646 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:37.773655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.774632 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.774640 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:37.776355 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:37.776366 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.777489 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.777494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:37.777654 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:37.777677 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:09:37.777790 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 22:09:37.777915 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904,"mutex_wait_us":71,"thread_start_us":447,"threads_started":1}
I20260927 22:09:37.778810 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:37.771668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39595 (local address 127.0.97.2:46247)
0927 22:09:37.772768 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:37.772782 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:37.772930 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:37.773237 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:37.773241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:37.773267 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:09:37.773653 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:37.773661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.774637 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.774642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:37.776469 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:37.776481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.777620 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.777626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:37.777782 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:37.777804 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 22:09:37.777909 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 22:09:37.778032 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":996,"mutex_wait_us":77,"thread_start_us":553,"threads_started":1}
I20260927 22:09:37.779357 8134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb86a7786e44fbaba65c09c56a485b2" candidate_uuid: "7f91d96917494a90aa3515a177f867e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9055928a5514436eb0d1b85c292a7762" is_pre_election: true
I20260927 22:09:37.779688 8416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb86a7786e44fbaba65c09c56a485b2" candidate_uuid: "7f91d96917494a90aa3515a177f867e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e458c806684e1fbe62ed518c3cbc23" is_pre_election: true
I20260927 22:09:37.779851 8134 raft_consensus.cc:2454] T 9cb86a7786e44fbaba65c09c56a485b2 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f91d96917494a90aa3515a177f867e3 in term 0.
I20260927 22:09:37.780149 8416 raft_consensus.cc:2454] T 9cb86a7786e44fbaba65c09c56a485b2 P 06e458c806684e1fbe62ed518c3cbc23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f91d96917494a90aa3515a177f867e3 in term 0.
I20260927 22:09:37.780483 8203 leader_election.cc:302] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f91d96917494a90aa3515a177f867e3, 9055928a5514436eb0d1b85c292a7762; no voters:
I20260927 22:09:37.780853 8492 raft_consensus.cc:2781] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:09:37.780992 8492 raft_consensus.cc:482] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:09:37.781119 8492 raft_consensus.cc:3037] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:37.783411 8492 raft_consensus.cc:504] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } } peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } }
I20260927 22:09:37.784050 8492 leader_election.cc:288] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3 [CANDIDATE]: Term 1 election: Requested vote from peers 06e458c806684e1fbe62ed518c3cbc23 (127.0.97.3:39595), 9055928a5514436eb0d1b85c292a7762 (127.0.97.1:38497)
I20260927 22:09:37.784417 8416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb86a7786e44fbaba65c09c56a485b2" candidate_uuid: "7f91d96917494a90aa3515a177f867e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e458c806684e1fbe62ed518c3cbc23"
I20260927 22:09:37.784619 8416 raft_consensus.cc:3037] T 9cb86a7786e44fbaba65c09c56a485b2 P 06e458c806684e1fbe62ed518c3cbc23 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:37.784580 8134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb86a7786e44fbaba65c09c56a485b2" candidate_uuid: "7f91d96917494a90aa3515a177f867e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9055928a5514436eb0d1b85c292a7762"
I20260927 22:09:37.784806 8134 raft_consensus.cc:3037] T 9cb86a7786e44fbaba65c09c56a485b2 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:37.787003 8134 raft_consensus.cc:2454] T 9cb86a7786e44fbaba65c09c56a485b2 P 9055928a5514436eb0d1b85c292a7762 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f91d96917494a90aa3515a177f867e3 in term 1.
I20260927 22:09:37.787039 8416 raft_consensus.cc:2454] T 9cb86a7786e44fbaba65c09c56a485b2 P 06e458c806684e1fbe62ed518c3cbc23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f91d96917494a90aa3515a177f867e3 in term 1.
I20260927 22:09:37.787407 8203 leader_election.cc:302] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f91d96917494a90aa3515a177f867e3, 9055928a5514436eb0d1b85c292a7762; no voters:
I20260927 22:09:37.787719 8492 raft_consensus.cc:2781] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:09:37.788093 8492 raft_consensus.cc:686] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3 [term 1 LEADER]: Becoming Leader. State: Replica: 7f91d96917494a90aa3515a177f867e3, State: Running, Role: LEADER
I20260927 22:09:37.788559 8492 consensus_queue.cc:237] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } } peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } }
I20260927 22:09:37.793111 7976 catalog_manager.cc:5495] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f91d96917494a90aa3515a177f867e3 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "7f91d96917494a90aa3515a177f867e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } health_report { overall_health: UNKNOWN } } }
I20260927 22:09:37.816671 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:37.811357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38974 (local address 127.0.97.2:45337)
0927 22:09:37.811546 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:37.811552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:37.811774 (+ 222us) server_negotiation.cc:407] Connection header received
0927 22:09:37.811993 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:37.811997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:37.812081 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:37.812225 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:09:37.812537 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.813223 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.814561 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:37.815035 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:37.815144 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:37.815548 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:09:37.816126 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:09:37.816239 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:37.816313 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 22:09:37.816375 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 22:09:37.824616 8134 raft_consensus.cc:1261] T 9cb86a7786e44fbaba65c09c56a485b2 P 9055928a5514436eb0d1b85c292a7762 [term 1 FOLLOWER]: Refusing update from remote peer 7f91d96917494a90aa3515a177f867e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:09:37.824674 8416 raft_consensus.cc:1261] T 9cb86a7786e44fbaba65c09c56a485b2 P 06e458c806684e1fbe62ed518c3cbc23 [term 1 FOLLOWER]: Refusing update from remote peer 7f91d96917494a90aa3515a177f867e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:09:37.825667 8496 consensus_queue.cc:1035] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:09:37.826035 8492 consensus_queue.cc:1035] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:09:37.857528 7976 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:09:37.857824 7976 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:09:37.862295 7976 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a: Sending DeleteTablet for 3 replicas of tablet 9cb86a7786e44fbaba65c09c56a485b2
I20260927 22:09:37.863509 8252 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cb86a7786e44fbaba65c09c56a485b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:37 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38968
I20260927 22:09:37.865427 8391 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cb86a7786e44fbaba65c09c56a485b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:37 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50978
I20260927 22:09:37.865942 8113 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cb86a7786e44fbaba65c09c56a485b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:37 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35776
I20260927 22:09:37.866496 8390 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cb86a7786e44fbaba65c09c56a485b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:37 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50978
I20260927 22:09:37.867573 7974 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:09:37.868374 7974 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:09:37.877107 8251 tablet_service.cc:1467] Processing CreateTablet for tablet a3f9fa0d5c04472b81603c07802cd816 (DEFAULT_TABLE table=test-table [id=18e589e86d8145638d82e229015b231d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:09:37.877230 8112 tablet_service.cc:1467] Processing CreateTablet for tablet a3f9fa0d5c04472b81603c07802cd816 (DEFAULT_TABLE table=test-table [id=18e589e86d8145638d82e229015b231d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:09:37.877707 8251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3f9fa0d5c04472b81603c07802cd816. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:37.877849 8112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3f9fa0d5c04472b81603c07802cd816. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:37.883210 8501 mvcc.cc:205] Tried to move back new op lower bound from 7334080421159886848 to 7334080421025918976. Current Snapshot: MvccSnapshot[applied={T|T < 7334080421159886848}]
I20260927 22:09:37.883448 8390 tablet_service.cc:1467] Processing CreateTablet for tablet a3f9fa0d5c04472b81603c07802cd816 (DEFAULT_TABLE table=test-table [id=18e589e86d8145638d82e229015b231d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:09:37.883941 8390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3f9fa0d5c04472b81603c07802cd816. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:37.885354 8485 tablet_bootstrap.cc:492] T a3f9fa0d5c04472b81603c07802cd816 P 7f91d96917494a90aa3515a177f867e3: Bootstrap starting.
I20260927 22:09:37.887121 8485 tablet_bootstrap.cc:654] T a3f9fa0d5c04472b81603c07802cd816 P 7f91d96917494a90aa3515a177f867e3: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:37.889566 8507 tablet_replica.cc:330] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3: stopping tablet replica
I20260927 22:09:37.890275 8507 raft_consensus.cc:2229] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:09:37.891198 8487 tablet_bootstrap.cc:492] T a3f9fa0d5c04472b81603c07802cd816 P 06e458c806684e1fbe62ed518c3cbc23: Bootstrap starting.
I20260927 22:09:37.891992 8507 raft_consensus.cc:2258] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:09:37.892374 8508 tablet_replica.cc:330] T 9cb86a7786e44fbaba65c09c56a485b2 P 06e458c806684e1fbe62ed518c3cbc23: stopping tablet replica
I20260927 22:09:37.893015 8487 tablet_bootstrap.cc:654] T a3f9fa0d5c04472b81603c07802cd816 P 06e458c806684e1fbe62ed518c3cbc23: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:37.893821 8486 tablet_bootstrap.cc:492] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762: Bootstrap starting.
I20260927 22:09:37.894080 8509 tablet_replica.cc:330] T 9cb86a7786e44fbaba65c09c56a485b2 P 9055928a5514436eb0d1b85c292a7762: stopping tablet replica
I20260927 22:09:37.894905 8509 raft_consensus.cc:2229] T 9cb86a7786e44fbaba65c09c56a485b2 P 9055928a5514436eb0d1b85c292a7762 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:09:37.895402 8509 raft_consensus.cc:2258] T 9cb86a7786e44fbaba65c09c56a485b2 P 9055928a5514436eb0d1b85c292a7762 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:09:37.895495 8486 tablet_bootstrap.cc:654] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:37.898010 8509 ts_tablet_manager.cc:1905] T 9cb86a7786e44fbaba65c09c56a485b2 P 9055928a5514436eb0d1b85c292a7762: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:37.901577 8508 raft_consensus.cc:2229] T 9cb86a7786e44fbaba65c09c56a485b2 P 06e458c806684e1fbe62ed518c3cbc23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:09:37.902498 8508 raft_consensus.cc:2258] T 9cb86a7786e44fbaba65c09c56a485b2 P 06e458c806684e1fbe62ed518c3cbc23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:09:37.904510 8486 tablet_bootstrap.cc:492] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762: No bootstrap required, opened a new log
I20260927 22:09:37.904711 8486 ts_tablet_manager.cc:1397] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20260927 22:09:37.905524 8486 raft_consensus.cc:348] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:37.905751 8486 raft_consensus.cc:374] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:37.905828 8486 raft_consensus.cc:729] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9055928a5514436eb0d1b85c292a7762, State: Initialized, Role: FOLLOWER
I20260927 22:09:37.906102 8486 consensus_queue.cc:260] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:37.906562 8485 tablet_bootstrap.cc:492] T a3f9fa0d5c04472b81603c07802cd816 P 7f91d96917494a90aa3515a177f867e3: No bootstrap required, opened a new log
I20260927 22:09:37.906800 8485 ts_tablet_manager.cc:1397] T a3f9fa0d5c04472b81603c07802cd816 P 7f91d96917494a90aa3515a177f867e3: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.005s
I20260927 22:09:37.907186 8507 ts_tablet_manager.cc:1905] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:37.907662 8485 raft_consensus.cc:348] T a3f9fa0d5c04472b81603c07802cd816 P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:37.907898 8485 raft_consensus.cc:374] T a3f9fa0d5c04472b81603c07802cd816 P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:37.908004 8485 raft_consensus.cc:729] T a3f9fa0d5c04472b81603c07802cd816 P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f91d96917494a90aa3515a177f867e3, State: Initialized, Role: FOLLOWER
I20260927 22:09:37.908277 8486 ts_tablet_manager.cc:1428] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 22:09:37.908288 8485 consensus_queue.cc:260] T a3f9fa0d5c04472b81603c07802cd816 P 7f91d96917494a90aa3515a177f867e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:37.910012 8485 ts_tablet_manager.cc:1428] T a3f9fa0d5c04472b81603c07802cd816 P 7f91d96917494a90aa3515a177f867e3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 22:09:37.912575 8508 ts_tablet_manager.cc:1905] T 9cb86a7786e44fbaba65c09c56a485b2 P 06e458c806684e1fbe62ed518c3cbc23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:37.913048 8509 ts_tablet_manager.cc:1918] T 9cb86a7786e44fbaba65c09c56a485b2 P 9055928a5514436eb0d1b85c292a7762: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:09:37.913236 8509 log.cc:1196] T 9cb86a7786e44fbaba65c09c56a485b2 P 9055928a5514436eb0d1b85c292a7762: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-0/wal/wals/9cb86a7786e44fbaba65c09c56a485b2
I20260927 22:09:37.913779 8509 ts_tablet_manager.cc:1939] T 9cb86a7786e44fbaba65c09c56a485b2 P 9055928a5514436eb0d1b85c292a7762: Deleting consensus metadata
I20260927 22:09:37.914251 8487 tablet_bootstrap.cc:492] T a3f9fa0d5c04472b81603c07802cd816 P 06e458c806684e1fbe62ed518c3cbc23: No bootstrap required, opened a new log
I20260927 22:09:37.914427 8487 ts_tablet_manager.cc:1397] T a3f9fa0d5c04472b81603c07802cd816 P 06e458c806684e1fbe62ed518c3cbc23: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.006s
I20260927 22:09:37.915158 8487 raft_consensus.cc:348] T a3f9fa0d5c04472b81603c07802cd816 P 06e458c806684e1fbe62ed518c3cbc23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:37.915366 8487 raft_consensus.cc:374] T a3f9fa0d5c04472b81603c07802cd816 P 06e458c806684e1fbe62ed518c3cbc23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:37.915472 8487 raft_consensus.cc:729] T a3f9fa0d5c04472b81603c07802cd816 P 06e458c806684e1fbe62ed518c3cbc23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e458c806684e1fbe62ed518c3cbc23, State: Initialized, Role: FOLLOWER
I20260927 22:09:37.915642 7960 catalog_manager.cc:4841] TS 9055928a5514436eb0d1b85c292a7762 (127.0.97.1:38497): tablet 9cb86a7786e44fbaba65c09c56a485b2 (table test-table [id=1e7e5758a56c4559a0d9ed0e101a26bd]) successfully deleted
I20260927 22:09:37.915761 8487 consensus_queue.cc:260] T a3f9fa0d5c04472b81603c07802cd816 P 06e458c806684e1fbe62ed518c3cbc23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:37.916352 8487 ts_tablet_manager.cc:1428] T a3f9fa0d5c04472b81603c07802cd816 P 06e458c806684e1fbe62ed518c3cbc23: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260927 22:09:37.919708 8508 ts_tablet_manager.cc:1918] T 9cb86a7786e44fbaba65c09c56a485b2 P 06e458c806684e1fbe62ed518c3cbc23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:09:37.919840 8508 log.cc:1196] T 9cb86a7786e44fbaba65c09c56a485b2 P 06e458c806684e1fbe62ed518c3cbc23: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/9cb86a7786e44fbaba65c09c56a485b2
I20260927 22:09:37.920343 8508 ts_tablet_manager.cc:1939] T 9cb86a7786e44fbaba65c09c56a485b2 P 06e458c806684e1fbe62ed518c3cbc23: Deleting consensus metadata
I20260927 22:09:37.921710 7962 catalog_manager.cc:4841] TS 06e458c806684e1fbe62ed518c3cbc23 (127.0.97.3:39595): tablet 9cb86a7786e44fbaba65c09c56a485b2 (table test-table [id=1e7e5758a56c4559a0d9ed0e101a26bd]) successfully deleted
W20260927 22:09:37.922205 7962 catalog_manager.cc:4808] TS 06e458c806684e1fbe62ed518c3cbc23 (127.0.97.3:39595): delete failed for tablet 9cb86a7786e44fbaba65c09c56a485b2 because the tablet was not found. No further retry: Not found: Tablet not found: 9cb86a7786e44fbaba65c09c56a485b2
I20260927 22:09:37.928500 8507 ts_tablet_manager.cc:1918] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:09:37.928696 8507 log.cc:1196] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/9cb86a7786e44fbaba65c09c56a485b2
I20260927 22:09:37.929242 8507 ts_tablet_manager.cc:1939] T 9cb86a7786e44fbaba65c09c56a485b2 P 7f91d96917494a90aa3515a177f867e3: Deleting consensus metadata
I20260927 22:09:37.931039 7960 catalog_manager.cc:4841] TS 7f91d96917494a90aa3515a177f867e3 (127.0.97.2:45337): tablet 9cb86a7786e44fbaba65c09c56a485b2 (table test-table [id=1e7e5758a56c4559a0d9ed0e101a26bd]) successfully deleted
I20260927 22:09:38.092937 8493 raft_consensus.cc:482] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:09:38.093149 8493 raft_consensus.cc:504] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:38.094928 8493 leader_election.cc:288] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f91d96917494a90aa3515a177f867e3 (127.0.97.2:45337), 06e458c806684e1fbe62ed518c3cbc23 (127.0.97.3:39595)
I20260927 22:09:38.103145 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:38.095363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50275 (local address 127.0.97.3:39595)
0927 22:09:38.095614 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:38.095623 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:38.096661 (+ 1038us) server_negotiation.cc:407] Connection header received
0927 22:09:38.096826 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:38.096834 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:38.096918 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:38.097008 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:09:38.097671 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.098827 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.101291 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.102106 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.102262 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:38.102361 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:38.102682 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 22:09:38.102804 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260927 22:09:38.103320 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:38.095352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39595 (local address 127.0.97.1:50275)
0927 22:09:38.096549 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:38.096566 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:38.096695 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:38.097067 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:38.097071 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:38.097095 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:09:38.097488 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:38.097499 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.098948 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.098954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:38.101152 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:38.101161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.102319 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.102325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:38.102491 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:38.102514 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:09:38.102658 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 22:09:38.102787 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1063,"thread_start_us":856,"threads_started":1}
I20260927 22:09:38.104213 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:38.095644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60065 (local address 127.0.97.2:45337)
0927 22:09:38.095806 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:38.095817 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:38.096304 (+ 487us) server_negotiation.cc:407] Connection header received
0927 22:09:38.096634 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:38.096643 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:38.096736 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:38.096850 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:09:38.097472 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.098292 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.101102 (+ 2810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.102518 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.102728 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:38.102819 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:38.103754 (+ 935us) server_negotiation.cc:299] Negotiation successful
0927 22:09:38.103909 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:09:38.104285 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:38.095237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45337 (local address 127.0.97.1:60065)
0927 22:09:38.096172 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:38.096200 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:38.096453 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:38.096863 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:38.096868 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:38.096889 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:09:38.097283 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:38.097294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.098432 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.098441 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:38.100814 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:38.100830 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.103027 (+ 2197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.103033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:38.103492 (+ 459us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:38.103538 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 22:09:38.103674 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 22:09:38.103855 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":80,"thread_start_us":636,"threads_started":1}
I20260927 22:09:38.104286 8416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3f9fa0d5c04472b81603c07802cd816" candidate_uuid: "9055928a5514436eb0d1b85c292a7762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e458c806684e1fbe62ed518c3cbc23" is_pre_election: true
I20260927 22:09:38.104549 8416 raft_consensus.cc:2454] T a3f9fa0d5c04472b81603c07802cd816 P 06e458c806684e1fbe62ed518c3cbc23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9055928a5514436eb0d1b85c292a7762 in term 0.
I20260927 22:09:38.105051 8272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3f9fa0d5c04472b81603c07802cd816" candidate_uuid: "9055928a5514436eb0d1b85c292a7762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f91d96917494a90aa3515a177f867e3" is_pre_election: true
I20260927 22:09:38.105129 8065 leader_election.cc:302] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e458c806684e1fbe62ed518c3cbc23, 9055928a5514436eb0d1b85c292a7762; no voters:
I20260927 22:09:38.105516 8493 raft_consensus.cc:2781] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:09:38.105492 8272 raft_consensus.cc:2454] T a3f9fa0d5c04472b81603c07802cd816 P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9055928a5514436eb0d1b85c292a7762 in term 0.
I20260927 22:09:38.105639 8493 raft_consensus.cc:482] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:09:38.105736 8493 raft_consensus.cc:3037] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:38.108023 8493 raft_consensus.cc:504] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:38.108652 8493 leader_election.cc:288] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762 [CANDIDATE]: Term 1 election: Requested vote from peers 7f91d96917494a90aa3515a177f867e3 (127.0.97.2:45337), 06e458c806684e1fbe62ed518c3cbc23 (127.0.97.3:39595)
I20260927 22:09:38.108997 8272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3f9fa0d5c04472b81603c07802cd816" candidate_uuid: "9055928a5514436eb0d1b85c292a7762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f91d96917494a90aa3515a177f867e3"
I20260927 22:09:38.109205 8272 raft_consensus.cc:3037] T a3f9fa0d5c04472b81603c07802cd816 P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:38.109382 8416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3f9fa0d5c04472b81603c07802cd816" candidate_uuid: "9055928a5514436eb0d1b85c292a7762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e458c806684e1fbe62ed518c3cbc23"
I20260927 22:09:38.109601 8416 raft_consensus.cc:3037] T a3f9fa0d5c04472b81603c07802cd816 P 06e458c806684e1fbe62ed518c3cbc23 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:38.111816 8272 raft_consensus.cc:2454] T a3f9fa0d5c04472b81603c07802cd816 P 7f91d96917494a90aa3515a177f867e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9055928a5514436eb0d1b85c292a7762 in term 1.
I20260927 22:09:38.112273 8063 leader_election.cc:302] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f91d96917494a90aa3515a177f867e3, 9055928a5514436eb0d1b85c292a7762; no voters:
I20260927 22:09:38.112546 8493 raft_consensus.cc:2781] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:09:38.112525 8416 raft_consensus.cc:2454] T a3f9fa0d5c04472b81603c07802cd816 P 06e458c806684e1fbe62ed518c3cbc23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9055928a5514436eb0d1b85c292a7762 in term 1.
I20260927 22:09:38.113111 8493 raft_consensus.cc:686] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762 [term 1 LEADER]: Becoming Leader. State: Replica: 9055928a5514436eb0d1b85c292a7762, State: Running, Role: LEADER
I20260927 22:09:38.113569 8493 consensus_queue.cc:237] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:38.117304 7974 catalog_manager.cc:5495] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9055928a5514436eb0d1b85c292a7762 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9055928a5514436eb0d1b85c292a7762" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } health_report { overall_health: UNKNOWN } } }
Sep 27 22:09:38 dist-test-slave-mp03 krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546975, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:09:38.224327 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:38.201282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42709 (local address 127.0.0.1:56614)
0927 22:09:38.202809 (+ 1527us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:38.202833 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:38.203032 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:38.203595 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:38.203599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:38.204268 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:38.204443 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:38.204451 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.205528 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.205535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:38.206476 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:38.206494 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.207229 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.207235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:38.207405 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:38.215421 (+ 8016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:38.215485 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:38.218158 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:38.221575 (+ 3417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:38.221587 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:38.221596 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:38.221996 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:38.222370 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:38.222375 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:38.222380 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:38.222521 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:38.223305 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:38.223315 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:38.223546 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 22:09:38.223753 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 22:09:38.224033 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1336,"thread_start_us":1143,"threads_started":1}
I20260927 22:09:38.224666 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:38.201290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56614 (local address 127.0.97.62:42709)
0927 22:09:38.203214 (+ 1924us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:38.203222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:38.203249 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:09:38.203290 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:38.203328 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:38.203419 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:38.203573 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:38.204586 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.205382 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.206655 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.207076 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.207212 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:38.218399 (+ 11187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:38.218404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:38.218407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:38.218436 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:38.221422 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:38.222117 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:38.222125 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:38.222129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:38.222253 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:38.222657 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:38.222661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:38.222665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:38.223048 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:38.223232 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:38.223968 (+ 736us) server_negotiation.cc:299] Negotiation successful
0927 22:09:38.224205 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1795,"thread_start_us":1647,"threads_started":1}
I20260927 22:09:38.229830 7974 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56614:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 22:09:38.230986 7974 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:09:38.238433 8112 tablet_service.cc:1467] Processing CreateTablet for tablet f55575d1396f4cd4ad3b1b7799e9dfdd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=faf0c53e3d38444d982ab316f4609376]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:09:38.239068 8112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55575d1396f4cd4ad3b1b7799e9dfdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:38.238950 8252 tablet_service.cc:1467] Processing CreateTablet for tablet f55575d1396f4cd4ad3b1b7799e9dfdd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=faf0c53e3d38444d982ab316f4609376]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:09:38.239553 8252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55575d1396f4cd4ad3b1b7799e9dfdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:38.242202 8391 tablet_service.cc:1467] Processing CreateTablet for tablet f55575d1396f4cd4ad3b1b7799e9dfdd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=faf0c53e3d38444d982ab316f4609376]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:09:38.242888 8391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55575d1396f4cd4ad3b1b7799e9dfdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:38.247131 8486 tablet_bootstrap.cc:492] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762: Bootstrap starting.
I20260927 22:09:38.247148 8485 tablet_bootstrap.cc:492] T f55575d1396f4cd4ad3b1b7799e9dfdd P 7f91d96917494a90aa3515a177f867e3: Bootstrap starting.
I20260927 22:09:38.248986 8485 tablet_bootstrap.cc:654] T f55575d1396f4cd4ad3b1b7799e9dfdd P 7f91d96917494a90aa3515a177f867e3: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:38.248977 8486 tablet_bootstrap.cc:654] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:38.250914 8487 tablet_bootstrap.cc:492] T f55575d1396f4cd4ad3b1b7799e9dfdd P 06e458c806684e1fbe62ed518c3cbc23: Bootstrap starting.
I20260927 22:09:38.252570 8487 tablet_bootstrap.cc:654] T f55575d1396f4cd4ad3b1b7799e9dfdd P 06e458c806684e1fbe62ed518c3cbc23: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:38.259146 8487 tablet_bootstrap.cc:492] T f55575d1396f4cd4ad3b1b7799e9dfdd P 06e458c806684e1fbe62ed518c3cbc23: No bootstrap required, opened a new log
I20260927 22:09:38.259321 8487 ts_tablet_manager.cc:1397] T f55575d1396f4cd4ad3b1b7799e9dfdd P 06e458c806684e1fbe62ed518c3cbc23: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260927 22:09:38.260188 8487 raft_consensus.cc:348] T f55575d1396f4cd4ad3b1b7799e9dfdd P 06e458c806684e1fbe62ed518c3cbc23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:38.260452 8487 raft_consensus.cc:374] T f55575d1396f4cd4ad3b1b7799e9dfdd P 06e458c806684e1fbe62ed518c3cbc23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:38.260557 8487 raft_consensus.cc:729] T f55575d1396f4cd4ad3b1b7799e9dfdd P 06e458c806684e1fbe62ed518c3cbc23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e458c806684e1fbe62ed518c3cbc23, State: Initialized, Role: FOLLOWER
I20260927 22:09:38.261037 8486 tablet_bootstrap.cc:492] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762: No bootstrap required, opened a new log
I20260927 22:09:38.261217 8486 ts_tablet_manager.cc:1397] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260927 22:09:38.261579 8485 tablet_bootstrap.cc:492] T f55575d1396f4cd4ad3b1b7799e9dfdd P 7f91d96917494a90aa3515a177f867e3: No bootstrap required, opened a new log
I20260927 22:09:38.261795 8485 ts_tablet_manager.cc:1397] T f55575d1396f4cd4ad3b1b7799e9dfdd P 7f91d96917494a90aa3515a177f867e3: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.005s
I20260927 22:09:38.261778 8487 consensus_queue.cc:260] T f55575d1396f4cd4ad3b1b7799e9dfdd P 06e458c806684e1fbe62ed518c3cbc23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:38.261983 8486 raft_consensus.cc:348] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:38.262246 8486 raft_consensus.cc:374] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:38.262343 8486 raft_consensus.cc:729] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9055928a5514436eb0d1b85c292a7762, State: Initialized, Role: FOLLOWER
I20260927 22:09:38.262239 8491 tablet_replica.cc:439] T f55575d1396f4cd4ad3b1b7799e9dfdd P 06e458c806684e1fbe62ed518c3cbc23: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } } }
I20260927 22:09:38.262610 8491 tablet_replica.cc:442] T f55575d1396f4cd4ad3b1b7799e9dfdd P 06e458c806684e1fbe62ed518c3cbc23: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:09:38.263113 8485 raft_consensus.cc:348] T f55575d1396f4cd4ad3b1b7799e9dfdd P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:38.263082 8486 consensus_queue.cc:260] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:38.263449 8485 raft_consensus.cc:374] T f55575d1396f4cd4ad3b1b7799e9dfdd P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:38.263583 8485 raft_consensus.cc:729] T f55575d1396f4cd4ad3b1b7799e9dfdd P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f91d96917494a90aa3515a177f867e3, State: Initialized, Role: FOLLOWER
I20260927 22:09:38.263547 8493 tablet_replica.cc:439] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } } }
I20260927 22:09:38.263814 8493 tablet_replica.cc:442] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:09:38.264003 8485 consensus_queue.cc:260] T f55575d1396f4cd4ad3b1b7799e9dfdd P 7f91d96917494a90aa3515a177f867e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:38.264508 8492 tablet_replica.cc:439] T f55575d1396f4cd4ad3b1b7799e9dfdd P 7f91d96917494a90aa3515a177f867e3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } } }
I20260927 22:09:38.264768 8492 tablet_replica.cc:442] T f55575d1396f4cd4ad3b1b7799e9dfdd P 7f91d96917494a90aa3515a177f867e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:09:38.266408 8486 ts_tablet_manager.cc:1428] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260927 22:09:38.266494 8487 ts_tablet_manager.cc:1428] T f55575d1396f4cd4ad3b1b7799e9dfdd P 06e458c806684e1fbe62ed518c3cbc23: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20260927 22:09:38.267325 8485 ts_tablet_manager.cc:1428] T f55575d1396f4cd4ad3b1b7799e9dfdd P 7f91d96917494a90aa3515a177f867e3: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260927 22:09:38.423131 8493 raft_consensus.cc:482] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:09:38.423390 8493 raft_consensus.cc:504] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:38.424078 8493 leader_election.cc:288] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f91d96917494a90aa3515a177f867e3 (127.0.97.2:45337), 06e458c806684e1fbe62ed518c3cbc23 (127.0.97.3:39595)
I20260927 22:09:38.424633 8416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55575d1396f4cd4ad3b1b7799e9dfdd" candidate_uuid: "9055928a5514436eb0d1b85c292a7762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e458c806684e1fbe62ed518c3cbc23" is_pre_election: true
I20260927 22:09:38.424638 8272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55575d1396f4cd4ad3b1b7799e9dfdd" candidate_uuid: "9055928a5514436eb0d1b85c292a7762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f91d96917494a90aa3515a177f867e3" is_pre_election: true
I20260927 22:09:38.424866 8416 raft_consensus.cc:2454] T f55575d1396f4cd4ad3b1b7799e9dfdd P 06e458c806684e1fbe62ed518c3cbc23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9055928a5514436eb0d1b85c292a7762 in term 0.
I20260927 22:09:38.424927 8272 raft_consensus.cc:2454] T f55575d1396f4cd4ad3b1b7799e9dfdd P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9055928a5514436eb0d1b85c292a7762 in term 0.
I20260927 22:09:38.425309 8065 leader_election.cc:302] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e458c806684e1fbe62ed518c3cbc23, 9055928a5514436eb0d1b85c292a7762; no voters:
I20260927 22:09:38.425593 8493 raft_consensus.cc:2781] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:09:38.425674 8493 raft_consensus.cc:482] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:09:38.425805 8493 raft_consensus.cc:3037] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:38.427763 8493 raft_consensus.cc:504] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:38.428289 8493 leader_election.cc:288] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762 [CANDIDATE]: Term 1 election: Requested vote from peers 7f91d96917494a90aa3515a177f867e3 (127.0.97.2:45337), 06e458c806684e1fbe62ed518c3cbc23 (127.0.97.3:39595)
I20260927 22:09:38.428766 8272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55575d1396f4cd4ad3b1b7799e9dfdd" candidate_uuid: "9055928a5514436eb0d1b85c292a7762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f91d96917494a90aa3515a177f867e3"
I20260927 22:09:38.428777 8416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55575d1396f4cd4ad3b1b7799e9dfdd" candidate_uuid: "9055928a5514436eb0d1b85c292a7762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e458c806684e1fbe62ed518c3cbc23"
I20260927 22:09:38.428961 8272 raft_consensus.cc:3037] T f55575d1396f4cd4ad3b1b7799e9dfdd P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:38.429055 8416 raft_consensus.cc:3037] T f55575d1396f4cd4ad3b1b7799e9dfdd P 06e458c806684e1fbe62ed518c3cbc23 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:38.430871 8272 raft_consensus.cc:2454] T f55575d1396f4cd4ad3b1b7799e9dfdd P 7f91d96917494a90aa3515a177f867e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9055928a5514436eb0d1b85c292a7762 in term 1.
I20260927 22:09:38.431155 8416 raft_consensus.cc:2454] T f55575d1396f4cd4ad3b1b7799e9dfdd P 06e458c806684e1fbe62ed518c3cbc23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9055928a5514436eb0d1b85c292a7762 in term 1.
I20260927 22:09:38.431268 8063 leader_election.cc:302] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f91d96917494a90aa3515a177f867e3, 9055928a5514436eb0d1b85c292a7762; no voters:
I20260927 22:09:38.431537 8493 raft_consensus.cc:2781] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:09:38.431710 8493 raft_consensus.cc:686] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762 [term 1 LEADER]: Becoming Leader. State: Replica: 9055928a5514436eb0d1b85c292a7762, State: Running, Role: LEADER
I20260927 22:09:38.431988 8493 consensus_queue.cc:237] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:38.433332 8516 tablet_replica.cc:439] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762: TxnStatusTablet state changed. Reason: New leader 9055928a5514436eb0d1b85c292a7762. Latest consensus state: current_term: 1 leader_uuid: "9055928a5514436eb0d1b85c292a7762" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } } }
I20260927 22:09:38.433559 8516 tablet_replica.cc:442] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:09:38.434108 8529 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 22:09:38.434332 8529 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 22:09:38.434904 7976 catalog_manager.cc:5495] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9055928a5514436eb0d1b85c292a7762 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9055928a5514436eb0d1b85c292a7762" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } health_report { overall_health: UNKNOWN } } }
I20260927 22:09:38.457489 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:38.450410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38497 (local address 127.0.0.1:46362)
0927 22:09:38.450743 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:38.450769 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:38.450900 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:38.451482 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:38.451485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:38.451495 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:09:38.451644 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:38.451651 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.453262 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.453266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:38.454787 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:38.454818 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.455812 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.455816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:38.455946 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:38.455998 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:09:38.457087 (+ 1089us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:09:38.457101 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 22:09:38.457152 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 22:09:38.457257 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 22:09:38.457706 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:38.450373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46362 (local address 127.0.97.1:38497)
0927 22:09:38.451036 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:38.451042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:38.451062 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:09:38.451157 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:38.451162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:38.451287 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:38.451467 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:09:38.451770 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.453037 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.454972 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.455669 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.455817 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:38.456198 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:09:38.456987 (+ 789us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:09:38.457129 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:38.457175 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 22:09:38.457255 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":383,"threads_started":1}
I20260927 22:09:38.462419 8416 raft_consensus.cc:1261] T f55575d1396f4cd4ad3b1b7799e9dfdd P 06e458c806684e1fbe62ed518c3cbc23 [term 1 FOLLOWER]: Refusing update from remote peer 9055928a5514436eb0d1b85c292a7762: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:09:38.462571 8272 raft_consensus.cc:1261] T f55575d1396f4cd4ad3b1b7799e9dfdd P 7f91d96917494a90aa3515a177f867e3 [term 1 FOLLOWER]: Refusing update from remote peer 9055928a5514436eb0d1b85c292a7762: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:09:38.463260 8516 consensus_queue.cc:1035] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:09:38.463615 8493 consensus_queue.cc:1035] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:09:38.467196 8491 tablet_replica.cc:439] T f55575d1396f4cd4ad3b1b7799e9dfdd P 06e458c806684e1fbe62ed518c3cbc23: TxnStatusTablet state changed. Reason: New leader 9055928a5514436eb0d1b85c292a7762. Latest consensus state: current_term: 1 leader_uuid: "9055928a5514436eb0d1b85c292a7762" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } } }
I20260927 22:09:38.467490 8491 tablet_replica.cc:442] T f55575d1396f4cd4ad3b1b7799e9dfdd P 06e458c806684e1fbe62ed518c3cbc23: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:09:38.468886 8492 tablet_replica.cc:439] T f55575d1396f4cd4ad3b1b7799e9dfdd P 7f91d96917494a90aa3515a177f867e3: TxnStatusTablet state changed. Reason: New leader 9055928a5514436eb0d1b85c292a7762. Latest consensus state: current_term: 1 leader_uuid: "9055928a5514436eb0d1b85c292a7762" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } } }
I20260927 22:09:38.469131 8492 tablet_replica.cc:442] T f55575d1396f4cd4ad3b1b7799e9dfdd P 7f91d96917494a90aa3515a177f867e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:09:38.469563 8533 mvcc.cc:205] Tried to move back new op lower bound from 7334080423775105024 to 7334080423659696128. Current Snapshot: MvccSnapshot[applied={T|T < 7334080423775105024}]
I20260927 22:09:38.475608 8535 mvcc.cc:205] Tried to move back new op lower bound from 7334080423775105024 to 7334080423659696128. Current Snapshot: MvccSnapshot[applied={T|T < 7334080423775105024}]
I20260927 22:09:38.480224 8516 tablet_replica.cc:439] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9055928a5514436eb0d1b85c292a7762" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } } }
I20260927 22:09:38.480597 8516 tablet_replica.cc:442] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:09:38.482257 8516 tablet_replica.cc:439] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9055928a5514436eb0d1b85c292a7762" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } } }
I20260927 22:09:38.482519 8516 tablet_replica.cc:442] T f55575d1396f4cd4ad3b1b7799e9dfdd P 9055928a5514436eb0d1b85c292a7762: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:09:38.485112 8491 tablet_replica.cc:439] T f55575d1396f4cd4ad3b1b7799e9dfdd P 06e458c806684e1fbe62ed518c3cbc23: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9055928a5514436eb0d1b85c292a7762" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } } }
I20260927 22:09:38.485351 8491 tablet_replica.cc:442] T f55575d1396f4cd4ad3b1b7799e9dfdd P 06e458c806684e1fbe62ed518c3cbc23: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:09:38.489317 8492 tablet_replica.cc:439] T f55575d1396f4cd4ad3b1b7799e9dfdd P 7f91d96917494a90aa3515a177f867e3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9055928a5514436eb0d1b85c292a7762" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } } }
I20260927 22:09:38.489576 8492 tablet_replica.cc:442] T f55575d1396f4cd4ad3b1b7799e9dfdd P 7f91d96917494a90aa3515a177f867e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:09:38.496946 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:38.491575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46372 (local address 127.0.97.1:38497)
0927 22:09:38.491761 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:38.491767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:38.491909 (+ 142us) server_negotiation.cc:407] Connection header received
0927 22:09:38.492055 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:38.492061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:38.492141 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:38.492238 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:09:38.492613 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.493738 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.495090 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.495784 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.495928 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:38.496239 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:09:38.496360 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:09:38.496450 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:38.496541 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 22:09:38.496617 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:09:38.514194 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:38.507727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38497 (local address 127.0.97.1:54459)
0927 22:09:38.508458 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:38.508475 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:38.508589 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:38.508944 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:38.508947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:38.508956 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:09:38.509256 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:38.509264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.510719 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.510722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:38.512075 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:38.512086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.513099 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.513104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:38.513224 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:38.513274 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:09:38.513674 (+ 400us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:09:38.513682 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:09:38.513744 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 22:09:38.513855 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":398,"threads_started":1}
I20260927 22:09:38.514221 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:38.507701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54459 (local address 127.0.97.1:38497)
0927 22:09:38.507898 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:38.507905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:38.508557 (+ 652us) server_negotiation.cc:407] Connection header received
0927 22:09:38.508725 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:38.508731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:38.508823 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:38.508924 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:09:38.509406 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.510562 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.512198 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.512875 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.513025 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:38.513434 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:09:38.513566 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:09:38.513652 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:38.513796 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 22:09:38.513872 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":52}
I20260927 22:09:38.522946 8416 raft_consensus.cc:1261] T a3f9fa0d5c04472b81603c07802cd816 P 06e458c806684e1fbe62ed518c3cbc23 [term 1 FOLLOWER]: Refusing update from remote peer 9055928a5514436eb0d1b85c292a7762: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:09:38.523238 8271 raft_consensus.cc:1261] T a3f9fa0d5c04472b81603c07802cd816 P 7f91d96917494a90aa3515a177f867e3 [term 1 FOLLOWER]: Refusing update from remote peer 9055928a5514436eb0d1b85c292a7762: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:09:38.523707 8516 consensus_queue.cc:1035] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:09:38.524160 8493 consensus_queue.cc:1035] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:09:38.588517 7976 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:09:38.588779 7976 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:09:38.591827 7976 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a: Sending DeleteTablet for 3 replicas of tablet a3f9fa0d5c04472b81603c07802cd816
I20260927 22:09:38.592707 8252 tablet_service.cc:1514] Processing DeleteTablet for tablet a3f9fa0d5c04472b81603c07802cd816 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:38 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38968
I20260927 22:09:38.592733 8113 tablet_service.cc:1514] Processing DeleteTablet for tablet a3f9fa0d5c04472b81603c07802cd816 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:38 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35776
I20260927 22:09:38.593150 8391 tablet_service.cc:1514] Processing DeleteTablet for tablet a3f9fa0d5c04472b81603c07802cd816 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:38 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50978
I20260927 22:09:38.593379 8550 tablet_replica.cc:330] T a3f9fa0d5c04472b81603c07802cd816 P 7f91d96917494a90aa3515a177f867e3: stopping tablet replica
I20260927 22:09:38.593580 8550 raft_consensus.cc:2229] T a3f9fa0d5c04472b81603c07802cd816 P 7f91d96917494a90aa3515a177f867e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:09:38.593828 8550 raft_consensus.cc:2258] T a3f9fa0d5c04472b81603c07802cd816 P 7f91d96917494a90aa3515a177f867e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:09:38.594525 8552 tablet_replica.cc:330] T a3f9fa0d5c04472b81603c07802cd816 P 06e458c806684e1fbe62ed518c3cbc23: stopping tablet replica
I20260927 22:09:38.594800 8552 raft_consensus.cc:2229] T a3f9fa0d5c04472b81603c07802cd816 P 06e458c806684e1fbe62ed518c3cbc23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:09:38.595111 8552 raft_consensus.cc:2258] T a3f9fa0d5c04472b81603c07802cd816 P 06e458c806684e1fbe62ed518c3cbc23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:09:38.595605 8550 ts_tablet_manager.cc:1905] T a3f9fa0d5c04472b81603c07802cd816 P 7f91d96917494a90aa3515a177f867e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:38.596189 8552 ts_tablet_manager.cc:1905] T a3f9fa0d5c04472b81603c07802cd816 P 06e458c806684e1fbe62ed518c3cbc23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:38.603602 8550 ts_tablet_manager.cc:1918] T a3f9fa0d5c04472b81603c07802cd816 P 7f91d96917494a90aa3515a177f867e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:09:38.604166 8550 log.cc:1196] T a3f9fa0d5c04472b81603c07802cd816 P 7f91d96917494a90aa3515a177f867e3: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/a3f9fa0d5c04472b81603c07802cd816
I20260927 22:09:38.604352 8552 ts_tablet_manager.cc:1918] T a3f9fa0d5c04472b81603c07802cd816 P 06e458c806684e1fbe62ed518c3cbc23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:09:38.604493 8552 log.cc:1196] T a3f9fa0d5c04472b81603c07802cd816 P 06e458c806684e1fbe62ed518c3cbc23: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/a3f9fa0d5c04472b81603c07802cd816
I20260927 22:09:38.604835 8550 ts_tablet_manager.cc:1939] T a3f9fa0d5c04472b81603c07802cd816 P 7f91d96917494a90aa3515a177f867e3: Deleting consensus metadata
I20260927 22:09:38.605183 8552 ts_tablet_manager.cc:1939] T a3f9fa0d5c04472b81603c07802cd816 P 06e458c806684e1fbe62ed518c3cbc23: Deleting consensus metadata
I20260927 22:09:38.606083 7960 catalog_manager.cc:4841] TS 7f91d96917494a90aa3515a177f867e3 (127.0.97.2:45337): tablet a3f9fa0d5c04472b81603c07802cd816 (table test-table [id=18e589e86d8145638d82e229015b231d]) successfully deleted
I20260927 22:09:38.606534 7962 catalog_manager.cc:4841] TS 06e458c806684e1fbe62ed518c3cbc23 (127.0.97.3:39595): tablet a3f9fa0d5c04472b81603c07802cd816 (table test-table [id=18e589e86d8145638d82e229015b231d]) successfully deleted
I20260927 22:09:38.611032 8551 tablet_replica.cc:330] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762: stopping tablet replica
I20260927 22:09:38.611294 8551 raft_consensus.cc:2229] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:09:38.612006 8551 raft_consensus.cc:2258] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:09:38.613149 8551 ts_tablet_manager.cc:1905] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:38.621516 8551 ts_tablet_manager.cc:1918] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:09:38.621672 8551 log.cc:1196] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-0/wal/wals/a3f9fa0d5c04472b81603c07802cd816
I20260927 22:09:38.622116 8551 ts_tablet_manager.cc:1939] T a3f9fa0d5c04472b81603c07802cd816 P 9055928a5514436eb0d1b85c292a7762: Deleting consensus metadata
I20260927 22:09:38.623139 7960 catalog_manager.cc:4841] TS 9055928a5514436eb0d1b85c292a7762 (127.0.97.1:38497): tablet a3f9fa0d5c04472b81603c07802cd816 (table test-table [id=18e589e86d8145638d82e229015b231d]) successfully deleted
Sep 27 22:09:38 dist-test-slave-mp03 krb5kdc[7931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:09:38 dist-test-slave-mp03 krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:38 dist-test-slave-mp03 krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:09:38.658553 8523 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:38.647181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42660 (local address 127.0.97.62:42709)
0927 22:09:38.647384 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:38.647392 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:38.647880 (+ 488us) server_negotiation.cc:407] Connection header received
0927 22:09:38.648074 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:38.648078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:38.648157 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:38.648236 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:38.649889 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.650665 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.652484 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.652900 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.652991 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:38.658246 (+ 5255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42660: BlockingRecv error: recv got EOF from 127.0.0.1:42660 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 22:09:38 dist-test-slave-mp03 krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:09:38.675738 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:38.661727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.0.97.62:42709)
0927 22:09:38.661965 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:38.661974 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:38.662465 (+ 491us) server_negotiation.cc:407] Connection header received
0927 22:09:38.662701 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:38.662709 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:38.662777 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:38.662887 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:38.663973 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.664731 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.665961 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.666388 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.666525 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:38.670339 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:38.670345 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:38.670348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:38.670382 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:38.673074 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:38.673711 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:38.673717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:38.673721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:38.673862 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:38.674235 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:38.674241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:38.674244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:38.674541 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:38.674855 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:38.675142 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 22:09:38.675364 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:09:38.679365 7976 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:09:38.680064 7976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:09:38.686877 8113 tablet_service.cc:1467] Processing CreateTablet for tablet b936ba2c1f6943fa93b6db481014d539 (DEFAULT_TABLE table=test-table [id=d484ffa072fd4ce59f4eebdf8f5ae591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:09:38.687212 8391 tablet_service.cc:1467] Processing CreateTablet for tablet b936ba2c1f6943fa93b6db481014d539 (DEFAULT_TABLE table=test-table [id=d484ffa072fd4ce59f4eebdf8f5ae591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:09:38.687373 8113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b936ba2c1f6943fa93b6db481014d539. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:38.687341 8252 tablet_service.cc:1467] Processing CreateTablet for tablet b936ba2c1f6943fa93b6db481014d539 (DEFAULT_TABLE table=test-table [id=d484ffa072fd4ce59f4eebdf8f5ae591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:09:38.687635 8391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b936ba2c1f6943fa93b6db481014d539. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:38.687700 8252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b936ba2c1f6943fa93b6db481014d539. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:38.694031 8487 tablet_bootstrap.cc:492] T b936ba2c1f6943fa93b6db481014d539 P 06e458c806684e1fbe62ed518c3cbc23: Bootstrap starting.
I20260927 22:09:38.694281 8486 tablet_bootstrap.cc:492] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762: Bootstrap starting.
I20260927 22:09:38.694568 8485 tablet_bootstrap.cc:492] T b936ba2c1f6943fa93b6db481014d539 P 7f91d96917494a90aa3515a177f867e3: Bootstrap starting.
I20260927 22:09:38.695811 8487 tablet_bootstrap.cc:654] T b936ba2c1f6943fa93b6db481014d539 P 06e458c806684e1fbe62ed518c3cbc23: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:38.695947 8486 tablet_bootstrap.cc:654] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:38.696213 8485 tablet_bootstrap.cc:654] T b936ba2c1f6943fa93b6db481014d539 P 7f91d96917494a90aa3515a177f867e3: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:38.710054 8485 tablet_bootstrap.cc:492] T b936ba2c1f6943fa93b6db481014d539 P 7f91d96917494a90aa3515a177f867e3: No bootstrap required, opened a new log
I20260927 22:09:38.710392 8485 ts_tablet_manager.cc:1397] T b936ba2c1f6943fa93b6db481014d539 P 7f91d96917494a90aa3515a177f867e3: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.010s
I20260927 22:09:38.712627 8486 tablet_bootstrap.cc:492] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762: No bootstrap required, opened a new log
I20260927 22:09:38.712831 8486 ts_tablet_manager.cc:1397] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.009s
I20260927 22:09:38.713586 8485 raft_consensus.cc:348] T b936ba2c1f6943fa93b6db481014d539 P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:38.713729 8486 raft_consensus.cc:348] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:38.714059 8486 raft_consensus.cc:374] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:38.714169 8486 raft_consensus.cc:729] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9055928a5514436eb0d1b85c292a7762, State: Initialized, Role: FOLLOWER
I20260927 22:09:38.714110 8485 raft_consensus.cc:374] T b936ba2c1f6943fa93b6db481014d539 P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:38.714352 8485 raft_consensus.cc:729] T b936ba2c1f6943fa93b6db481014d539 P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f91d96917494a90aa3515a177f867e3, State: Initialized, Role: FOLLOWER
I20260927 22:09:38.714432 8486 consensus_queue.cc:260] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:38.714787 8485 consensus_queue.cc:260] T b936ba2c1f6943fa93b6db481014d539 P 7f91d96917494a90aa3515a177f867e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:38.715031 8486 ts_tablet_manager.cc:1428] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:09:38.719322 8485 ts_tablet_manager.cc:1428] T b936ba2c1f6943fa93b6db481014d539 P 7f91d96917494a90aa3515a177f867e3: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20260927 22:09:38.721027 8487 tablet_bootstrap.cc:492] T b936ba2c1f6943fa93b6db481014d539 P 06e458c806684e1fbe62ed518c3cbc23: No bootstrap required, opened a new log
I20260927 22:09:38.721184 8487 ts_tablet_manager.cc:1397] T b936ba2c1f6943fa93b6db481014d539 P 06e458c806684e1fbe62ed518c3cbc23: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.010s
I20260927 22:09:38.721956 8487 raft_consensus.cc:348] T b936ba2c1f6943fa93b6db481014d539 P 06e458c806684e1fbe62ed518c3cbc23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:38.722218 8487 raft_consensus.cc:374] T b936ba2c1f6943fa93b6db481014d539 P 06e458c806684e1fbe62ed518c3cbc23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:38.722327 8487 raft_consensus.cc:729] T b936ba2c1f6943fa93b6db481014d539 P 06e458c806684e1fbe62ed518c3cbc23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e458c806684e1fbe62ed518c3cbc23, State: Initialized, Role: FOLLOWER
I20260927 22:09:38.722579 8487 consensus_queue.cc:260] T b936ba2c1f6943fa93b6db481014d539 P 06e458c806684e1fbe62ed518c3cbc23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:38.723187 8487 ts_tablet_manager.cc:1428] T b936ba2c1f6943fa93b6db481014d539 P 06e458c806684e1fbe62ed518c3cbc23: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 22:09:38.892202 8546 raft_consensus.cc:482] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:09:38.892436 8546 raft_consensus.cc:504] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:38.893358 8546 leader_election.cc:288] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f91d96917494a90aa3515a177f867e3 (127.0.97.2:45337), 06e458c806684e1fbe62ed518c3cbc23 (127.0.97.3:39595)
I20260927 22:09:38.893924 8416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b936ba2c1f6943fa93b6db481014d539" candidate_uuid: "9055928a5514436eb0d1b85c292a7762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e458c806684e1fbe62ed518c3cbc23" is_pre_election: true
I20260927 22:09:38.893942 8272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b936ba2c1f6943fa93b6db481014d539" candidate_uuid: "9055928a5514436eb0d1b85c292a7762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f91d96917494a90aa3515a177f867e3" is_pre_election: true
I20260927 22:09:38.894162 8416 raft_consensus.cc:2454] T b936ba2c1f6943fa93b6db481014d539 P 06e458c806684e1fbe62ed518c3cbc23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9055928a5514436eb0d1b85c292a7762 in term 0.
I20260927 22:09:38.894188 8272 raft_consensus.cc:2454] T b936ba2c1f6943fa93b6db481014d539 P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9055928a5514436eb0d1b85c292a7762 in term 0.
I20260927 22:09:38.894661 8063 leader_election.cc:302] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f91d96917494a90aa3515a177f867e3, 9055928a5514436eb0d1b85c292a7762; no voters:
I20260927 22:09:38.895005 8546 raft_consensus.cc:2781] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:09:38.895128 8546 raft_consensus.cc:482] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:09:38.895255 8546 raft_consensus.cc:3037] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:38.897596 8546 raft_consensus.cc:504] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:38.898391 8546 leader_election.cc:288] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762 [CANDIDATE]: Term 1 election: Requested vote from peers 7f91d96917494a90aa3515a177f867e3 (127.0.97.2:45337), 06e458c806684e1fbe62ed518c3cbc23 (127.0.97.3:39595)
I20260927 22:09:38.898711 8272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b936ba2c1f6943fa93b6db481014d539" candidate_uuid: "9055928a5514436eb0d1b85c292a7762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f91d96917494a90aa3515a177f867e3"
I20260927 22:09:38.898936 8272 raft_consensus.cc:3037] T b936ba2c1f6943fa93b6db481014d539 P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:38.898952 8416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b936ba2c1f6943fa93b6db481014d539" candidate_uuid: "9055928a5514436eb0d1b85c292a7762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e458c806684e1fbe62ed518c3cbc23"
I20260927 22:09:38.899145 8416 raft_consensus.cc:3037] T b936ba2c1f6943fa93b6db481014d539 P 06e458c806684e1fbe62ed518c3cbc23 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:38.901125 8416 raft_consensus.cc:2454] T b936ba2c1f6943fa93b6db481014d539 P 06e458c806684e1fbe62ed518c3cbc23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9055928a5514436eb0d1b85c292a7762 in term 1.
I20260927 22:09:38.901546 8065 leader_election.cc:302] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e458c806684e1fbe62ed518c3cbc23, 9055928a5514436eb0d1b85c292a7762; no voters:
I20260927 22:09:38.901701 8272 raft_consensus.cc:2454] T b936ba2c1f6943fa93b6db481014d539 P 7f91d96917494a90aa3515a177f867e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9055928a5514436eb0d1b85c292a7762 in term 1.
I20260927 22:09:38.901830 8546 raft_consensus.cc:2781] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:09:38.901975 8546 raft_consensus.cc:686] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762 [term 1 LEADER]: Becoming Leader. State: Replica: 9055928a5514436eb0d1b85c292a7762, State: Running, Role: LEADER
I20260927 22:09:38.902247 8546 consensus_queue.cc:237] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:38.905023 7975 catalog_manager.cc:5495] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9055928a5514436eb0d1b85c292a7762 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9055928a5514436eb0d1b85c292a7762" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } health_report { overall_health: UNKNOWN } } }
I20260927 22:09:38.956053 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:38.950700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46376 (local address 127.0.97.1:38497)
0927 22:09:38.950960 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:38.950965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:38.950982 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:09:38.951175 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:38.951179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:38.951250 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:38.951322 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:09:38.951731 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.952430 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.954345 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:38.954891 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:38.954992 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:38.955471 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:09:38.955573 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:09:38.955636 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:38.955790 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 22:09:38.955848 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260927 22:09:38.958838 8416 raft_consensus.cc:1261] T b936ba2c1f6943fa93b6db481014d539 P 06e458c806684e1fbe62ed518c3cbc23 [term 1 FOLLOWER]: Refusing update from remote peer 9055928a5514436eb0d1b85c292a7762: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:09:38.958902 8272 raft_consensus.cc:1261] T b936ba2c1f6943fa93b6db481014d539 P 7f91d96917494a90aa3515a177f867e3 [term 1 FOLLOWER]: Refusing update from remote peer 9055928a5514436eb0d1b85c292a7762: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:09:38.959497 8493 consensus_queue.cc:1035] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:09:38.959805 8546 consensus_queue.cc:1035] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:09:38.976559 7976 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:09:38.976825 7976 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:09:38.979790 7976 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a: Sending DeleteTablet for 3 replicas of tablet b936ba2c1f6943fa93b6db481014d539
I20260927 22:09:38.980546 8113 tablet_service.cc:1514] Processing DeleteTablet for tablet b936ba2c1f6943fa93b6db481014d539 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:38 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35776
I20260927 22:09:38.980698 8252 tablet_service.cc:1514] Processing DeleteTablet for tablet b936ba2c1f6943fa93b6db481014d539 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:38 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38968
I20260927 22:09:38.980820 8551 tablet_replica.cc:330] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762: stopping tablet replica
I20260927 22:09:38.981024 8551 raft_consensus.cc:2229] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:09:38.981343 8551 raft_consensus.cc:2258] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:09:38.981618 8550 tablet_replica.cc:330] T b936ba2c1f6943fa93b6db481014d539 P 7f91d96917494a90aa3515a177f867e3: stopping tablet replica
I20260927 22:09:38.981812 8550 raft_consensus.cc:2229] T b936ba2c1f6943fa93b6db481014d539 P 7f91d96917494a90aa3515a177f867e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:09:38.982017 8550 raft_consensus.cc:2258] T b936ba2c1f6943fa93b6db481014d539 P 7f91d96917494a90aa3515a177f867e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:09:38.982710 8391 tablet_service.cc:1514] Processing DeleteTablet for tablet b936ba2c1f6943fa93b6db481014d539 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:38 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50978
I20260927 22:09:38.982968 8552 tablet_replica.cc:330] T b936ba2c1f6943fa93b6db481014d539 P 06e458c806684e1fbe62ed518c3cbc23: stopping tablet replica
I20260927 22:09:38.983166 8552 raft_consensus.cc:2229] T b936ba2c1f6943fa93b6db481014d539 P 06e458c806684e1fbe62ed518c3cbc23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:09:38.983242 8550 ts_tablet_manager.cc:1905] T b936ba2c1f6943fa93b6db481014d539 P 7f91d96917494a90aa3515a177f867e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:38.983408 8552 raft_consensus.cc:2258] T b936ba2c1f6943fa93b6db481014d539 P 06e458c806684e1fbe62ed518c3cbc23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:09:38.987437 8552 ts_tablet_manager.cc:1905] T b936ba2c1f6943fa93b6db481014d539 P 06e458c806684e1fbe62ed518c3cbc23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:38.987443 8551 ts_tablet_manager.cc:1905] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:38.990108 8550 ts_tablet_manager.cc:1918] T b936ba2c1f6943fa93b6db481014d539 P 7f91d96917494a90aa3515a177f867e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:09:38.990250 8550 log.cc:1196] T b936ba2c1f6943fa93b6db481014d539 P 7f91d96917494a90aa3515a177f867e3: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/b936ba2c1f6943fa93b6db481014d539
I20260927 22:09:38.990681 8550 ts_tablet_manager.cc:1939] T b936ba2c1f6943fa93b6db481014d539 P 7f91d96917494a90aa3515a177f867e3: Deleting consensus metadata
I20260927 22:09:38.991802 7960 catalog_manager.cc:4841] TS 7f91d96917494a90aa3515a177f867e3 (127.0.97.2:45337): tablet b936ba2c1f6943fa93b6db481014d539 (table test-table [id=d484ffa072fd4ce59f4eebdf8f5ae591]) successfully deleted
I20260927 22:09:38.994093 8552 ts_tablet_manager.cc:1918] T b936ba2c1f6943fa93b6db481014d539 P 06e458c806684e1fbe62ed518c3cbc23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:09:38.994257 8552 log.cc:1196] T b936ba2c1f6943fa93b6db481014d539 P 06e458c806684e1fbe62ed518c3cbc23: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/b936ba2c1f6943fa93b6db481014d539
I20260927 22:09:38.994663 8552 ts_tablet_manager.cc:1939] T b936ba2c1f6943fa93b6db481014d539 P 06e458c806684e1fbe62ed518c3cbc23: Deleting consensus metadata
I20260927 22:09:38.995329 8551 ts_tablet_manager.cc:1918] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:09:38.995483 8551 log.cc:1196] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-0/wal/wals/b936ba2c1f6943fa93b6db481014d539
I20260927 22:09:38.995814 7962 catalog_manager.cc:4841] TS 06e458c806684e1fbe62ed518c3cbc23 (127.0.97.3:39595): tablet b936ba2c1f6943fa93b6db481014d539 (table test-table [id=d484ffa072fd4ce59f4eebdf8f5ae591]) successfully deleted
I20260927 22:09:38.995929 8551 ts_tablet_manager.cc:1939] T b936ba2c1f6943fa93b6db481014d539 P 9055928a5514436eb0d1b85c292a7762: Deleting consensus metadata
I20260927 22:09:38.997004 7960 catalog_manager.cc:4841] TS 9055928a5514436eb0d1b85c292a7762 (127.0.97.1:38497): tablet b936ba2c1f6943fa93b6db481014d539 (table test-table [id=d484ffa072fd4ce59f4eebdf8f5ae591]) successfully deleted
I20260927 22:09:38.997108 7976 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:09:38.997696 7976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:09:39.004400 8252 tablet_service.cc:1467] Processing CreateTablet for tablet 5834324bd8b4440c9e46e888cdfa8a86 (DEFAULT_TABLE table=test-table [id=d2b7d013222645c980048c8340d9e6b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:09:39.004374 8113 tablet_service.cc:1467] Processing CreateTablet for tablet 5834324bd8b4440c9e46e888cdfa8a86 (DEFAULT_TABLE table=test-table [id=d2b7d013222645c980048c8340d9e6b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:09:39.004755 8252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5834324bd8b4440c9e46e888cdfa8a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:39.004796 8113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5834324bd8b4440c9e46e888cdfa8a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:39.008828 8391 tablet_service.cc:1467] Processing CreateTablet for tablet 5834324bd8b4440c9e46e888cdfa8a86 (DEFAULT_TABLE table=test-table [id=d2b7d013222645c980048c8340d9e6b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:09:39.009272 8391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5834324bd8b4440c9e46e888cdfa8a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:09:39.009505 8485 tablet_bootstrap.cc:492] T 5834324bd8b4440c9e46e888cdfa8a86 P 7f91d96917494a90aa3515a177f867e3: Bootstrap starting.
I20260927 22:09:39.011106 8485 tablet_bootstrap.cc:654] T 5834324bd8b4440c9e46e888cdfa8a86 P 7f91d96917494a90aa3515a177f867e3: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:39.012261 8486 tablet_bootstrap.cc:492] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762: Bootstrap starting.
I20260927 22:09:39.014258 8486 tablet_bootstrap.cc:654] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:39.018888 8487 tablet_bootstrap.cc:492] T 5834324bd8b4440c9e46e888cdfa8a86 P 06e458c806684e1fbe62ed518c3cbc23: Bootstrap starting.
I20260927 22:09:39.020912 8487 tablet_bootstrap.cc:654] T 5834324bd8b4440c9e46e888cdfa8a86 P 06e458c806684e1fbe62ed518c3cbc23: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:39.021328 8485 tablet_bootstrap.cc:492] T 5834324bd8b4440c9e46e888cdfa8a86 P 7f91d96917494a90aa3515a177f867e3: No bootstrap required, opened a new log
I20260927 22:09:39.021488 8485 ts_tablet_manager.cc:1397] T 5834324bd8b4440c9e46e888cdfa8a86 P 7f91d96917494a90aa3515a177f867e3: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260927 22:09:39.022573 8485 raft_consensus.cc:348] T 5834324bd8b4440c9e46e888cdfa8a86 P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:39.022924 8485 raft_consensus.cc:374] T 5834324bd8b4440c9e46e888cdfa8a86 P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:39.022998 8485 raft_consensus.cc:729] T 5834324bd8b4440c9e46e888cdfa8a86 P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f91d96917494a90aa3515a177f867e3, State: Initialized, Role: FOLLOWER
I20260927 22:09:39.023332 8485 consensus_queue.cc:260] T 5834324bd8b4440c9e46e888cdfa8a86 P 7f91d96917494a90aa3515a177f867e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:39.024798 8486 tablet_bootstrap.cc:492] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762: No bootstrap required, opened a new log
I20260927 22:09:39.024961 8486 ts_tablet_manager.cc:1397] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20260927 22:09:39.024979 8485 ts_tablet_manager.cc:1428] T 5834324bd8b4440c9e46e888cdfa8a86 P 7f91d96917494a90aa3515a177f867e3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 22:09:39.025717 8486 raft_consensus.cc:348] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:39.025981 8486 raft_consensus.cc:374] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:39.026062 8486 raft_consensus.cc:729] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9055928a5514436eb0d1b85c292a7762, State: Initialized, Role: FOLLOWER
I20260927 22:09:39.026309 8486 consensus_queue.cc:260] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:39.026836 8486 ts_tablet_manager.cc:1428] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:09:39.032001 8487 tablet_bootstrap.cc:492] T 5834324bd8b4440c9e46e888cdfa8a86 P 06e458c806684e1fbe62ed518c3cbc23: No bootstrap required, opened a new log
I20260927 22:09:39.032644 8487 ts_tablet_manager.cc:1397] T 5834324bd8b4440c9e46e888cdfa8a86 P 06e458c806684e1fbe62ed518c3cbc23: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.007s
I20260927 22:09:39.033281 8487 raft_consensus.cc:348] T 5834324bd8b4440c9e46e888cdfa8a86 P 06e458c806684e1fbe62ed518c3cbc23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:39.033461 8487 raft_consensus.cc:374] T 5834324bd8b4440c9e46e888cdfa8a86 P 06e458c806684e1fbe62ed518c3cbc23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:39.033556 8487 raft_consensus.cc:729] T 5834324bd8b4440c9e46e888cdfa8a86 P 06e458c806684e1fbe62ed518c3cbc23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e458c806684e1fbe62ed518c3cbc23, State: Initialized, Role: FOLLOWER
I20260927 22:09:39.033819 8487 consensus_queue.cc:260] T 5834324bd8b4440c9e46e888cdfa8a86 P 06e458c806684e1fbe62ed518c3cbc23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:39.034314 8487 ts_tablet_manager.cc:1428] T 5834324bd8b4440c9e46e888cdfa8a86 P 06e458c806684e1fbe62ed518c3cbc23: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260927 22:09:39.049278 8516 raft_consensus.cc:482] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:09:39.049496 8516 raft_consensus.cc:504] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:39.050190 8516 leader_election.cc:288] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f91d96917494a90aa3515a177f867e3 (127.0.97.2:45337), 06e458c806684e1fbe62ed518c3cbc23 (127.0.97.3:39595)
I20260927 22:09:39.050530 8272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5834324bd8b4440c9e46e888cdfa8a86" candidate_uuid: "9055928a5514436eb0d1b85c292a7762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f91d96917494a90aa3515a177f867e3" is_pre_election: true
I20260927 22:09:39.050800 8272 raft_consensus.cc:2454] T 5834324bd8b4440c9e46e888cdfa8a86 P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9055928a5514436eb0d1b85c292a7762 in term 0.
I20260927 22:09:39.050866 8416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5834324bd8b4440c9e46e888cdfa8a86" candidate_uuid: "9055928a5514436eb0d1b85c292a7762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e458c806684e1fbe62ed518c3cbc23" is_pre_election: true
I20260927 22:09:39.051110 8416 raft_consensus.cc:2454] T 5834324bd8b4440c9e46e888cdfa8a86 P 06e458c806684e1fbe62ed518c3cbc23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9055928a5514436eb0d1b85c292a7762 in term 0.
I20260927 22:09:39.051179 8063 leader_election.cc:302] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f91d96917494a90aa3515a177f867e3, 9055928a5514436eb0d1b85c292a7762; no voters:
I20260927 22:09:39.051461 8516 raft_consensus.cc:2781] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:09:39.051600 8516 raft_consensus.cc:482] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:09:39.051726 8516 raft_consensus.cc:3037] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:39.053787 8516 raft_consensus.cc:504] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:39.054370 8516 leader_election.cc:288] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762 [CANDIDATE]: Term 1 election: Requested vote from peers 7f91d96917494a90aa3515a177f867e3 (127.0.97.2:45337), 06e458c806684e1fbe62ed518c3cbc23 (127.0.97.3:39595)
I20260927 22:09:39.054757 8272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5834324bd8b4440c9e46e888cdfa8a86" candidate_uuid: "9055928a5514436eb0d1b85c292a7762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f91d96917494a90aa3515a177f867e3"
I20260927 22:09:39.054869 8416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5834324bd8b4440c9e46e888cdfa8a86" candidate_uuid: "9055928a5514436eb0d1b85c292a7762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e458c806684e1fbe62ed518c3cbc23"
I20260927 22:09:39.054972 8272 raft_consensus.cc:3037] T 5834324bd8b4440c9e46e888cdfa8a86 P 7f91d96917494a90aa3515a177f867e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:39.055068 8416 raft_consensus.cc:3037] T 5834324bd8b4440c9e46e888cdfa8a86 P 06e458c806684e1fbe62ed518c3cbc23 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:39.057904 8272 raft_consensus.cc:2454] T 5834324bd8b4440c9e46e888cdfa8a86 P 7f91d96917494a90aa3515a177f867e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9055928a5514436eb0d1b85c292a7762 in term 1.
I20260927 22:09:39.057924 8416 raft_consensus.cc:2454] T 5834324bd8b4440c9e46e888cdfa8a86 P 06e458c806684e1fbe62ed518c3cbc23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9055928a5514436eb0d1b85c292a7762 in term 1.
I20260927 22:09:39.058324 8063 leader_election.cc:302] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f91d96917494a90aa3515a177f867e3, 9055928a5514436eb0d1b85c292a7762; no voters:
I20260927 22:09:39.058674 8516 raft_consensus.cc:2781] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:09:39.058951 8516 raft_consensus.cc:686] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762 [term 1 LEADER]: Becoming Leader. State: Replica: 9055928a5514436eb0d1b85c292a7762, State: Running, Role: LEADER
I20260927 22:09:39.059340 8516 consensus_queue.cc:237] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } }
I20260927 22:09:39.061849 7975 catalog_manager.cc:5495] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9055928a5514436eb0d1b85c292a7762 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9055928a5514436eb0d1b85c292a7762" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055928a5514436eb0d1b85c292a7762" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 } health_report { overall_health: UNKNOWN } } }
I20260927 22:09:39.099858 8272 raft_consensus.cc:1261] T 5834324bd8b4440c9e46e888cdfa8a86 P 7f91d96917494a90aa3515a177f867e3 [term 1 FOLLOWER]: Refusing update from remote peer 9055928a5514436eb0d1b85c292a7762: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:09:39.099897 8416 raft_consensus.cc:1261] T 5834324bd8b4440c9e46e888cdfa8a86 P 06e458c806684e1fbe62ed518c3cbc23 [term 1 FOLLOWER]: Refusing update from remote peer 9055928a5514436eb0d1b85c292a7762: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:09:39.100607 8516 consensus_queue.cc:1035] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f91d96917494a90aa3515a177f867e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:09:39.100859 8493 consensus_queue.cc:1035] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e458c806684e1fbe62ed518c3cbc23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:09:39.156741 7976 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:09:39.156988 7976 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:09:39.159837 7976 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4f0ca4681ee418fb8f5d8926467ac7a: Sending DeleteTablet for 3 replicas of tablet 5834324bd8b4440c9e46e888cdfa8a86
I20260927 22:09:39.160813 8391 tablet_service.cc:1514] Processing DeleteTablet for tablet 5834324bd8b4440c9e46e888cdfa8a86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:39 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50978
I20260927 22:09:39.161123 8113 tablet_service.cc:1514] Processing DeleteTablet for tablet 5834324bd8b4440c9e46e888cdfa8a86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:39 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35776
I20260927 22:09:39.161142 8252 tablet_service.cc:1514] Processing DeleteTablet for tablet 5834324bd8b4440c9e46e888cdfa8a86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:09:39 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38968
I20260927 22:09:39.161397 8551 tablet_replica.cc:330] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762: stopping tablet replica
I20260927 22:09:39.161439 8550 tablet_replica.cc:330] T 5834324bd8b4440c9e46e888cdfa8a86 P 7f91d96917494a90aa3515a177f867e3: stopping tablet replica
I20260927 22:09:39.161449 8552 tablet_replica.cc:330] T 5834324bd8b4440c9e46e888cdfa8a86 P 06e458c806684e1fbe62ed518c3cbc23: stopping tablet replica
I20260927 22:09:39.161576 8551 raft_consensus.cc:2229] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:09:39.161609 8552 raft_consensus.cc:2229] T 5834324bd8b4440c9e46e888cdfa8a86 P 06e458c806684e1fbe62ed518c3cbc23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:09:39.161623 8550 raft_consensus.cc:2229] T 5834324bd8b4440c9e46e888cdfa8a86 P 7f91d96917494a90aa3515a177f867e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:09:39.161811 8550 raft_consensus.cc:2258] T 5834324bd8b4440c9e46e888cdfa8a86 P 7f91d96917494a90aa3515a177f867e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:09:39.161811 8552 raft_consensus.cc:2258] T 5834324bd8b4440c9e46e888cdfa8a86 P 06e458c806684e1fbe62ed518c3cbc23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:09:39.161911 8551 raft_consensus.cc:2258] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:09:39.163067 8550 ts_tablet_manager.cc:1905] T 5834324bd8b4440c9e46e888cdfa8a86 P 7f91d96917494a90aa3515a177f867e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:39.163082 8551 ts_tablet_manager.cc:1905] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:39.163067 8552 ts_tablet_manager.cc:1905] T 5834324bd8b4440c9e46e888cdfa8a86 P 06e458c806684e1fbe62ed518c3cbc23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:09:39.167667 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 8047
I20260927 22:09:39.170131 8551 ts_tablet_manager.cc:1918] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:09:39.170287 8551 log.cc:1196] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-0/wal/wals/5834324bd8b4440c9e46e888cdfa8a86
I20260927 22:09:39.170398 8550 ts_tablet_manager.cc:1918] T 5834324bd8b4440c9e46e888cdfa8a86 P 7f91d96917494a90aa3515a177f867e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:09:39.170544 8550 log.cc:1196] T 5834324bd8b4440c9e46e888cdfa8a86 P 7f91d96917494a90aa3515a177f867e3: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/5834324bd8b4440c9e46e888cdfa8a86
I20260927 22:09:39.170778 8551 ts_tablet_manager.cc:1939] T 5834324bd8b4440c9e46e888cdfa8a86 P 9055928a5514436eb0d1b85c292a7762: Deleting consensus metadata
I20260927 22:09:39.170975 8550 ts_tablet_manager.cc:1939] T 5834324bd8b4440c9e46e888cdfa8a86 P 7f91d96917494a90aa3515a177f867e3: Deleting consensus metadata
I20260927 22:09:39.171896 7960 catalog_manager.cc:4841] TS 9055928a5514436eb0d1b85c292a7762 (127.0.97.1:38497): tablet 5834324bd8b4440c9e46e888cdfa8a86 (table test-table [id=d2b7d013222645c980048c8340d9e6b8]) successfully deleted
I20260927 22:09:39.172426 7960 catalog_manager.cc:4841] TS 7f91d96917494a90aa3515a177f867e3 (127.0.97.2:45337): tablet 5834324bd8b4440c9e46e888cdfa8a86 (table test-table [id=d2b7d013222645c980048c8340d9e6b8]) successfully deleted
I20260927 22:09:39.175168 8552 ts_tablet_manager.cc:1918] T 5834324bd8b4440c9e46e888cdfa8a86 P 06e458c806684e1fbe62ed518c3cbc23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:09:39.175356 8552 log.cc:1196] T 5834324bd8b4440c9e46e888cdfa8a86 P 06e458c806684e1fbe62ed518c3cbc23: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/5834324bd8b4440c9e46e888cdfa8a86
I20260927 22:09:39.175809 8552 ts_tablet_manager.cc:1939] T 5834324bd8b4440c9e46e888cdfa8a86 P 06e458c806684e1fbe62ed518c3cbc23: Deleting consensus metadata
I20260927 22:09:39.176852 7962 catalog_manager.cc:4841] TS 06e458c806684e1fbe62ed518c3cbc23 (127.0.97.3:39595): tablet 5834324bd8b4440c9e46e888cdfa8a86 (table test-table [id=d2b7d013222645c980048c8340d9e6b8]) successfully deleted
Sep 27 22:09:39 dist-test-slave-mp03 krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260927 22:09:39.191767 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:39.169108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46384 (local address 127.0.97.1:38497)
0927 22:09:39.169433 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:39.169442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:39.175913 (+ 6471us) server_negotiation.cc:407] Connection header received
0927 22:09:39.176193 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:39.176198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:39.176284 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:39.176413 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:39.177968 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:39.178837 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:39.180667 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:39.181105 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:39.181197 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:39.185773 (+ 4576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:39.185792 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:39.185809 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:39.185848 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:39.188928 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:39.189597 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:39.189608 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:39.189613 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:39.189784 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:39.190148 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:39.190154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:39.190159 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:39.190497 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:39.190832 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:39.191151 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 22:09:39.191375 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192}
I20260927 22:09:39.192418 8178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:39.338636 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 8047
I20260927 22:09:39.362883 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 8187
Sep 27 22:09:39 dist-test-slave-mp03 krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260927 22:09:39.377631 8582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:39.363652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42016 (local address 127.0.97.2:45337)
0927 22:09:39.364141 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:39.364145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:39.364160 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:09:39.364208 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:39.364211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:39.364273 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:39.364386 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:39.365549 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:39.366324 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:39.367623 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:39.368044 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:39.368171 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:39.372150 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:39.372170 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:39.372186 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:39.372217 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:39.375070 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:39.375676 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:39.375684 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:39.375687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:39.375799 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:39.376158 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:39.376164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:39.376167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:39.376501 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:39.376815 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:39.377085 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 22:09:39.377290 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":242,"threads_started":1}
I20260927 22:09:39.378291 8317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:39.496721 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 8187
I20260927 22:09:39.518280 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 8326
Sep 27 22:09:39 dist-test-slave-mp03 krb5kdc[7931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260927 22:09:39.532496 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:39.519082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47924 (local address 127.0.97.3:39595)
0927 22:09:39.519610 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:39.519614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:39.519630 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:09:39.519684 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:39.519688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:39.519745 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:39.519835 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:39.521061 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:39.521745 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:39.523086 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:39.523487 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:39.523577 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:39.527603 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:39.527615 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:39.527632 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:39.527662 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:39.530178 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:39.530904 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:39.530911 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:39.530914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:39.531024 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:39.531322 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:39.531324 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:39.531326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:39.531654 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:39.531860 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:39.532085 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 22:09:39.532236 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":310,"threads_started":1}
I20260927 22:09:39.533143 8456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:39.628755 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 8326
I20260927 22:09:39.645944 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 7944
I20260927 22:09:39.647136 8016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:39.759251 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 7944
2026-09-27T22:09:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5860 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:09:39 dist-test-slave-mp03 krb5kdc[8589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:09:39 dist-test-slave-mp03 krb5kdc[8589](info): set up 2 sockets
Sep 27 22:09:39 dist-test-slave-mp03 krb5kdc[8589](info): commencing operation
krb5kdc: starting...
W20260927 22:09:40.863730 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:09:41 dist-test-slave-mp03 krb5kdc[8589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:09:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:09:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:09:41.084460 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36733
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:41.403193 8602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:41.403524 8602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:41.403683 8602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:41.410838 8602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:09:41.410984 8602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:41.411090 8602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:41.411175 8602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:09:41.411262 8602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:09:41.421008 8602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:41.422351 8602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:41.423676 8602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:41.436036 8607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:41.436091 8608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:41.436091 8610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:41.436689 8602 server_base.cc:1034] running on GCE node
I20260927 22:09:41.437539 8602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:41.439013 8602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:41.440248 8602 hybrid_clock.cc:648] HybridClock initialized: now 1790546981440205 us; error 53 us; skew 500 ppm
I20260927 22:09:41.440593 8602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T22:09:41Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1751 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:09:41 dist-test-slave-mp03 krb5kdc[8616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:09:41 dist-test-slave-mp03 krb5kdc[8616](info): set up 2 sockets
Sep 27 22:09:41 dist-test-slave-mp03 krb5kdc[8616](info): commencing operation
krb5kdc: starting...
W20260927 22:09:42.607635 388 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:09:42 dist-test-slave-mp03 krb5kdc[8616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:09:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:09:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:09:42.830142 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44471
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:43.160451 8629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:43.160771 8629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:43.160928 8629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:43.168249 8629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:09:43.168371 8629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:43.168465 8629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:43.168567 8629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:09:43.168659 8629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:09:43.178777 8629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:43.180217 8629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:43.181582 8629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:43.193881 8634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:43.193965 8635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:43.193912 8637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:43.195425 8629 server_base.cc:1034] running on GCE node
I20260927 22:09:43.196350 8629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:43.197866 8629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:43.199103 8629 hybrid_clock.cc:648] HybridClock initialized: now 1790546983199059 us; error 62 us; skew 500 ppm
I20260927 22:09:43.199458 8629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:43 dist-test-slave-mp03 krb5kdc[8616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:43.202726 8629 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:09:43.204376 8629 webserver.cc:460] Webserver started at http://127.0.97.62:37637/ using document root <none> and password file <none>
I20260927 22:09:43.205087 8629 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:43.205233 8629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:43.205561 8629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:43.208581 8629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "ea5fcd9bf32b468c9c695460cd2ee05b"
format_stamp: "Formatted at 2026-09-27 22:09:43 on dist-test-slave-mp03"
server_key: "7734617e8d9c81931636c2d0961ac44e"
server_key_iv: "ca0a58df7ee2b2e090ea75c26e20a147"
server_key_version: "encryptionkey@0"
I20260927 22:09:43.209352 8629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "ea5fcd9bf32b468c9c695460cd2ee05b"
format_stamp: "Formatted at 2026-09-27 22:09:43 on dist-test-slave-mp03"
server_key: "7734617e8d9c81931636c2d0961ac44e"
server_key_iv: "ca0a58df7ee2b2e090ea75c26e20a147"
server_key_version: "encryptionkey@0"
I20260927 22:09:43.214787 8629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:09:43.218300 8644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:43.219755 8629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:09:43.219938 8629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "ea5fcd9bf32b468c9c695460cd2ee05b"
format_stamp: "Formatted at 2026-09-27 22:09:43 on dist-test-slave-mp03"
server_key: "7734617e8d9c81931636c2d0961ac44e"
server_key_iv: "ca0a58df7ee2b2e090ea75c26e20a147"
server_key_version: "encryptionkey@0"
I20260927 22:09:43.220134 8629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:43.246872 8629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:43.251219 8629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:43.251528 8629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:43.269620 8629 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44471
I20260927 22:09:43.269652 8695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44471 every 8 connection(s)
I20260927 22:09:43.271138 8629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:09:43.272567 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 8629
I20260927 22:09:43.272774 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:09:43.273593 388 external_mini_cluster.cc:1428] Setting key 5d1e4b54a7b6abb93c1ce8fabc30ee64
I20260927 22:09:43.276250 8696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:09:43 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:09:43.289358 8696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b: Bootstrap starting.
I20260927 22:09:43.292665 8696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b: Neither blocks nor log segments found. Creating new log.
I20260927 22:09:43.293712 8696 log.cc:824] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:43.296612 8699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:43.275706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60548 (local address 127.0.97.62:44471)
0927 22:09:43.276497 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:43.276508 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:43.276552 (+ 44us) server_negotiation.cc:407] Connection header received
0927 22:09:43.278159 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:43.278180 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:43.278552 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:43.278957 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:43.280751 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:43.282146 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:43.283887 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:43.284523 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:43.284653 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:43.289239 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:43.289247 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:43.289262 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:43.289302 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:43.292421 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:43.293200 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:43.293206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:43.293213 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:43.293372 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:43.293759 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:43.293768 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:43.293770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:43.294374 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:43.294721 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:43.295204 (+ 483us) server_negotiation.cc:299] Negotiation successful
0927 22:09:43.295608 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":361,"threads_started":1}
I20260927 22:09:43.297158 8696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b: No bootstrap required, opened a new log
I20260927 22:09:43.305791 8696 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5fcd9bf32b468c9c695460cd2ee05b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44471 } }
I20260927 22:09:43.306239 8696 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:09:43.306357 8696 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea5fcd9bf32b468c9c695460cd2ee05b, State: Initialized, Role: FOLLOWER
I20260927 22:09:43.307008 8696 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5fcd9bf32b468c9c695460cd2ee05b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44471 } }
I20260927 22:09:43.307258 8696 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:09:43.307417 8696 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:09:43.307600 8696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:09:43.310251 8696 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5fcd9bf32b468c9c695460cd2ee05b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44471 } }
I20260927 22:09:43.310822 8696 leader_election.cc:302] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea5fcd9bf32b468c9c695460cd2ee05b; no voters:
I20260927 22:09:43.311415 8696 leader_election.cc:288] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:09:43.311518 8701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:09:43.312242 8701 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [term 1 LEADER]: Becoming Leader. State: Replica: ea5fcd9bf32b468c9c695460cd2ee05b, State: Running, Role: LEADER
I20260927 22:09:43.312894 8701 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5fcd9bf32b468c9c695460cd2ee05b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44471 } }
I20260927 22:09:43.313922 8696 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:09:43.315624 8703 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea5fcd9bf32b468c9c695460cd2ee05b. Latest consensus state: current_term: 1 leader_uuid: "ea5fcd9bf32b468c9c695460cd2ee05b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5fcd9bf32b468c9c695460cd2ee05b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44471 } } }
I20260927 22:09:43.315819 8702 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea5fcd9bf32b468c9c695460cd2ee05b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5fcd9bf32b468c9c695460cd2ee05b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44471 } } }
I20260927 22:09:43.316115 8702 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:43.316126 8703 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:43.319954 8707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:09:43.327111 8707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:09:43.336215 8707 catalog_manager.cc:1347] Generated new cluster ID: 6ef12b037bfa42e59adeff11c66a289e
I20260927 22:09:43.336349 8707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:09:43.375092 8707 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:09:43.376601 8707 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:09:43.384634 8707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b: Generated new TSK 0
I20260927 22:09:43.385527 8707 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:09:43.451005 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44471
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:43.783830 8723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:43.784121 8723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:43.784307 8723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:43.791850 8723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:43.791961 8723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:43.792128 8723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:09:43.802380 8723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:43.803755 8723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:43.805044 8723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:43.818223 8729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:43.818442 8731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:43.818809 8728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:43.819315 8723 server_base.cc:1034] running on GCE node
I20260927 22:09:43.819970 8723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:43.821107 8723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:43.822409 8723 hybrid_clock.cc:648] HybridClock initialized: now 1790546983822330 us; error 59 us; skew 500 ppm
I20260927 22:09:43.822808 8723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:43 dist-test-slave-mp03 krb5kdc[8616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:43.826581 8723 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:09:43.828181 8723 webserver.cc:460] Webserver started at http://127.0.97.1:43493/ using document root <none> and password file <none>
I20260927 22:09:43.828902 8723 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:43.829048 8723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:43.829370 8723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:43.832315 8723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "191deb2a46de44248184307a6a68ebc0"
format_stamp: "Formatted at 2026-09-27 22:09:43 on dist-test-slave-mp03"
server_key: "a8cc9622622ce49aca0cd358c4760b8f"
server_key_iv: "7bff1964afebd6aa187353666660499f"
server_key_version: "encryptionkey@0"
I20260927 22:09:43.833076 8723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "191deb2a46de44248184307a6a68ebc0"
format_stamp: "Formatted at 2026-09-27 22:09:43 on dist-test-slave-mp03"
server_key: "a8cc9622622ce49aca0cd358c4760b8f"
server_key_iv: "7bff1964afebd6aa187353666660499f"
server_key_version: "encryptionkey@0"
I20260927 22:09:43.838135 8723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 22:09:43.841887 8738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:43.843349 8723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:09:43.843524 8723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "191deb2a46de44248184307a6a68ebc0"
format_stamp: "Formatted at 2026-09-27 22:09:43 on dist-test-slave-mp03"
server_key: "a8cc9622622ce49aca0cd358c4760b8f"
server_key_iv: "7bff1964afebd6aa187353666660499f"
server_key_version: "encryptionkey@0"
I20260927 22:09:43.843742 8723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:43.868067 8723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:43.872532 8723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:43.872839 8723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:43.873814 8723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:43.875164 8723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:43.875279 8723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:43.875429 8723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:43.875500 8723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:43.908502 8723 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41701
I20260927 22:09:43.908555 8850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41701 every 8 connection(s)
I20260927 22:09:43.909806 8723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:09:43.915823 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 8723
I20260927 22:09:43.916050 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:09:43.916685 388 external_mini_cluster.cc:1428] Setting key 82e6bc084806ceb0e026f972ee5c21a5
Sep 27 22:09:43 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:09:43.934479 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:43.912434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35133 (local address 127.0.97.62:44471)
0927 22:09:43.912888 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:43.912893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:43.914813 (+ 1920us) server_negotiation.cc:407] Connection header received
0927 22:09:43.917095 (+ 2282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:43.917101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:43.917221 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:43.917349 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:43.919362 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:43.920038 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:43.922646 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:43.923065 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:43.923259 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:43.928249 (+ 4990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:43.928253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:43.928257 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:43.928288 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:43.931055 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:43.932039 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:43.932043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:43.932046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:43.932166 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:43.932711 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:43.932715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:43.932717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:43.932969 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:43.933073 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:43.934038 (+ 965us) server_negotiation.cc:299] Negotiation successful
0927 22:09:43.934221 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":222,"threads_started":1}
I20260927 22:09:43.936120 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:43.913039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.1:35133)
0927 22:09:43.914513 (+ 1474us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:43.914584 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:43.916753 (+ 2169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:43.917632 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:43.917673 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:43.918269 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:43.919159 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:43.919190 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:43.920211 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:43.920220 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:43.922427 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:43.922444 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:43.923309 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:43.923319 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:43.923655 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:43.924560 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:43.924594 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:43.927984 (+ 3390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:43.931206 (+ 3222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:43.931218 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:43.931240 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:43.931900 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:43.932327 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:43.932335 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:43.932342 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:43.932591 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:43.933098 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:43.933109 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:43.933592 (+ 483us) client_negotiation.cc:769] Sending connection context
0927 22:09:43.933924 (+ 332us) client_negotiation.cc:240] Negotiation successful
0927 22:09:43.934518 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":364,"threads_started":1}
I20260927 22:09:43.937999 8851 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44471
I20260927 22:09:43.938421 8851 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:43.939464 8851 heartbeater.cc:511] Master 127.0.97.62:44471 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:09:43.941967 8661 ts_manager.cc:194] Registered new tserver with Master: 191deb2a46de44248184307a6a68ebc0 (127.0.97.1:41701)
I20260927 22:09:43.943657 8661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:09:43.971328 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44471
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:44.296218 8857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:44.296491 8857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:44.296583 8857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:44.303747 8857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:44.303861 8857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:44.303974 8857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:09:44.313577 8857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:44.314988 8857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:44.316395 8857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:44.330484 8862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:44.330439 8865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:44.330365 8863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:44.331179 8857 server_base.cc:1034] running on GCE node
I20260927 22:09:44.331770 8857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:44.332729 8857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:44.333971 8857 hybrid_clock.cc:648] HybridClock initialized: now 1790546984333901 us; error 58 us; skew 500 ppm
I20260927 22:09:44.334312 8857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:44 dist-test-slave-mp03 krb5kdc[8616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:44.337468 8857 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:09:44.338955 8857 webserver.cc:460] Webserver started at http://127.0.97.2:32929/ using document root <none> and password file <none>
I20260927 22:09:44.339633 8857 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:44.339762 8857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:44.340086 8857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:44.343075 8857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "eafe04d23a224b55a5a634d29e59669f"
format_stamp: "Formatted at 2026-09-27 22:09:44 on dist-test-slave-mp03"
server_key: "6162766aa132f2889640268b87341669"
server_key_iv: "c527fc730d6c2c941c15336969e28488"
server_key_version: "encryptionkey@0"
I20260927 22:09:44.343858 8857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "eafe04d23a224b55a5a634d29e59669f"
format_stamp: "Formatted at 2026-09-27 22:09:44 on dist-test-slave-mp03"
server_key: "6162766aa132f2889640268b87341669"
server_key_iv: "c527fc730d6c2c941c15336969e28488"
server_key_version: "encryptionkey@0"
I20260927 22:09:44.349036 8857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:09:44.352349 8872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:44.353973 8857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:09:44.354184 8857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "eafe04d23a224b55a5a634d29e59669f"
format_stamp: "Formatted at 2026-09-27 22:09:44 on dist-test-slave-mp03"
server_key: "6162766aa132f2889640268b87341669"
server_key_iv: "c527fc730d6c2c941c15336969e28488"
server_key_version: "encryptionkey@0"
I20260927 22:09:44.354432 8857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:44.386255 8857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:44.390424 8857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:44.390735 8857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:44.391650 8857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:44.392889 8857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:44.392967 8857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:44.393059 8857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:44.393108 8857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:44.426386 8857 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46503
I20260927 22:09:44.426433 8984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46503 every 8 connection(s)
I20260927 22:09:44.427727 8857 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:09:44.435009 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 8857
I20260927 22:09:44.435200 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:09:44.435806 388 external_mini_cluster.cc:1428] Setting key 4b485c408b18d8a2bc6a0ca1ad1e3c43
Sep 27 22:09:44 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:09:44.448963 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:44.430238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53399 (local address 127.0.97.62:44471)
0927 22:09:44.430438 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:44.430444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:44.431650 (+ 1206us) server_negotiation.cc:407] Connection header received
0927 22:09:44.433460 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:44.433464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:44.433531 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:44.433640 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:44.435651 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:44.436438 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:44.438656 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:44.439065 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:44.439163 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:44.443372 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:44.443376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:44.443379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:44.443409 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:44.445912 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:44.446666 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:44.446673 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:44.446681 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:44.446798 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:44.447241 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:44.447246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:44.447249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:44.447496 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:44.447648 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:44.448392 (+ 744us) server_negotiation.cc:299] Negotiation successful
0927 22:09:44.448612 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:09:44.450024 8987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:44.430809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.2:53399)
0927 22:09:44.431496 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:44.431533 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:44.433218 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:44.433914 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:44.433923 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:44.434473 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:44.435473 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:44.435500 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:44.436679 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:44.436685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:44.438437 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:44.438452 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:44.439306 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:44.439315 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:44.439696 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:44.440439 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:44.440475 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:44.443145 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:44.446054 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:44.446064 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:44.446082 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:44.446494 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:44.446908 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:44.446914 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:44.446919 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:44.447126 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:44.447627 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:44.447635 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:44.447981 (+ 346us) client_negotiation.cc:769] Sending connection context
0927 22:09:44.448303 (+ 322us) client_negotiation.cc:240] Negotiation successful
0927 22:09:44.448851 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":288,"threads_started":1}
I20260927 22:09:44.451660 8985 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44471
I20260927 22:09:44.451968 8985 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:44.452729 8985 heartbeater.cc:511] Master 127.0.97.62:44471 requested a full tablet report, sending...
I20260927 22:09:44.454578 8661 ts_manager.cc:194] Registered new tserver with Master: eafe04d23a224b55a5a634d29e59669f (127.0.97.2:46503)
I20260927 22:09:44.455559 8661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53399
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:09:44.490800 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44471
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:44.815393 8990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:44.815658 8990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:44.815814 8990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:44.823447 8990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:44.823627 8990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:44.823823 8990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:09:44.833811 8990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:44.835193 8990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:44.836527 8990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:44.849108 8995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:44.849308 8996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:44.849736 8998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:44.850464 8990 server_base.cc:1034] running on GCE node
I20260927 22:09:44.851055 8990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:44.851949 8990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:44.853144 8990 hybrid_clock.cc:648] HybridClock initialized: now 1790546984853088 us; error 54 us; skew 500 ppm
I20260927 22:09:44.853451 8990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:44 dist-test-slave-mp03 krb5kdc[8616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:44.856675 8990 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:09:44.858126 8990 webserver.cc:460] Webserver started at http://127.0.97.3:41897/ using document root <none> and password file <none>
I20260927 22:09:44.858940 8990 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:44.859082 8990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:44.859383 8990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:09:44.862262 8990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "6e982a0d66b94fd2bd0ffefb5c930cfd"
format_stamp: "Formatted at 2026-09-27 22:09:44 on dist-test-slave-mp03"
server_key: "8f0033be597a7856f5fba572bfda686b"
server_key_iv: "e25602adce3c013f5581806d6a1e6de5"
server_key_version: "encryptionkey@0"
I20260927 22:09:44.863056 8990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6e982a0d66b94fd2bd0ffefb5c930cfd"
format_stamp: "Formatted at 2026-09-27 22:09:44 on dist-test-slave-mp03"
server_key: "8f0033be597a7856f5fba572bfda686b"
server_key_iv: "e25602adce3c013f5581806d6a1e6de5"
server_key_version: "encryptionkey@0"
I20260927 22:09:44.867947 8990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 22:09:44.871340 9005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:44.872676 8990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:09:44.872944 8990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "6e982a0d66b94fd2bd0ffefb5c930cfd"
format_stamp: "Formatted at 2026-09-27 22:09:44 on dist-test-slave-mp03"
server_key: "8f0033be597a7856f5fba572bfda686b"
server_key_iv: "e25602adce3c013f5581806d6a1e6de5"
server_key_version: "encryptionkey@0"
I20260927 22:09:44.873145 8990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:44.890398 8990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:44.894434 8990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:44.894785 8990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:44.895726 8990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:44.897044 8990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:44.897149 8990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:44.897308 8990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:44.897382 8990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:44.931010 8990 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36523
I20260927 22:09:44.931054 9117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36523 every 8 connection(s)
I20260927 22:09:44.932341 8990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:09:44.941846 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 8990
I20260927 22:09:44.942013 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:09:44.942540 388 external_mini_cluster.cc:1428] Setting key a52a19947350527cdfd18f5895f04241
Sep 27 22:09:44 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:09:44.946782 8851 heartbeater.cc:503] Master 127.0.97.62:44471 was elected leader, sending a full tablet report...
I20260927 22:09:44.960122 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:44.935009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37843 (local address 127.0.97.62:44471)
0927 22:09:44.935177 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:44.935181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:44.936375 (+ 1194us) server_negotiation.cc:407] Connection header received
0927 22:09:44.938219 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:44.938223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:44.938293 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:44.938392 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:44.939984 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:44.940725 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:44.942936 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:44.943392 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:44.943513 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:44.947542 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:44.947548 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:44.947551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:44.947583 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:44.956753 (+ 9170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:44.957640 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:44.957645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:44.957650 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:44.957779 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:44.958264 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:44.958269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:44.958272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:44.958533 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:44.958797 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:44.959531 (+ 734us) server_negotiation.cc:299] Negotiation successful
0927 22:09:44.959786 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:09:44.961174 9120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:44.935487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.3:37843)
0927 22:09:44.936191 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:44.936235 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:44.937897 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:44.938659 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:44.938668 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:44.939220 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:44.939830 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:44.939848 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:44.940915 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:44.940919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:44.942749 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:44.942764 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:44.943634 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:44.943640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:44.943929 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:44.944718 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:44.944754 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:44.947325 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:09:44.956909 (+ 9584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:44.956919 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:44.956941 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:44.957508 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:44.957897 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:09:44.957901 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:09:44.957905 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:09:44.958139 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:09:44.958772 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:09:44.958781 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:09:44.959138 (+ 357us) client_negotiation.cc:769] Sending connection context
0927 22:09:44.959418 (+ 280us) client_negotiation.cc:240] Negotiation successful
0927 22:09:44.960024 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":292,"threads_started":1}
I20260927 22:09:44.962541 9118 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44471
I20260927 22:09:44.962883 9118 heartbeater.cc:464] Registering TS with master...
I20260927 22:09:44.963551 9118 heartbeater.cc:511] Master 127.0.97.62:44471 requested a full tablet report, sending...
I20260927 22:09:44.964887 8661 ts_manager.cc:194] Registered new tserver with Master: 6e982a0d66b94fd2bd0ffefb5c930cfd (127.0.97.3:36523)
I20260927 22:09:44.965674 8661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37843
I20260927 22:09:44.970108 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260927 22:09:45.113445 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 8723
Sep 27 22:09:45 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:09:45.130549 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:45.114713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37936 (local address 127.0.97.1:41701)
0927 22:09:45.115451 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:45.115459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:45.115486 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:09:45.115544 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:45.115550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:45.115770 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:45.115919 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:45.117281 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:45.118361 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:45.119687 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:45.120272 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:45.120434 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:45.124686 (+ 4252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:45.124699 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:45.124715 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:45.124756 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:45.127739 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:45.128446 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:45.128453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:45.128456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:45.128587 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:45.128964 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:45.128974 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:45.128976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:45.129323 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:45.129609 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:45.129924 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 22:09:45.130200 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":416,"threads_started":1}
I20260927 22:09:45.131844 8846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:45.237927 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 8723
I20260927 22:09:45.254107 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 8857
Sep 27 22:09:45 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:09:45.270088 9130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:45.255023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47804 (local address 127.0.97.2:46503)
0927 22:09:45.255508 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:45.255515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:45.255538 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:09:45.255595 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:45.255602 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:45.255813 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:45.255978 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:45.257208 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:45.258176 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:45.259714 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:45.260263 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:45.260406 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:45.264442 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:45.264454 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:45.264469 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:45.264503 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:45.267538 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:45.268141 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:45.268148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:45.268150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:45.268258 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:45.268545 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:45.268549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:45.268550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:45.268911 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:45.269209 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:45.269399 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 22:09:45.269658 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":256,"threads_started":1}
I20260927 22:09:45.271441 8980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:45.353188 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 8857
I20260927 22:09:45.373893 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 8990
Sep 27 22:09:45 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:09:45.390664 9132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:45.375087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48102 (local address 127.0.97.3:36523)
0927 22:09:45.375679 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:45.375686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:45.375719 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:09:45.375784 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:45.375791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:45.376013 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:45.376252 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:45.377452 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:45.378804 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:45.380045 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:45.380626 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:45.380728 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:45.385041 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:45.385053 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:45.385069 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:45.385121 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:45.387986 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:45.388731 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:45.388738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:45.388740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:45.388853 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:45.389192 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:45.389195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:45.389197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:45.389518 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:45.389721 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:45.390004 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 22:09:45.390200 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":284,"threads_started":1}
I20260927 22:09:45.392133 9113 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:45.475396 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 8990
I20260927 22:09:45.490850 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 8629
I20260927 22:09:45.492014 8691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:09:45.593432 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 8629
I20260927 22:09:45.613960 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44471
--webserver_interface=127.0.97.62
--webserver_port=37637
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:09:45.948051 9135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:45.948459 9135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:45.948644 9135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:45.956164 9135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:09:45.956350 9135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:45.956475 9135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:45.956597 9135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:09:45.956704 9135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:09:45.966938 9135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=37637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:45.968369 9135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:45.969753 9135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:45.982072 9140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:45.982097 9141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:45.982126 9143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:45.982774 9135 server_base.cc:1034] running on GCE node
I20260927 22:09:45.983583 9135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:45.985066 9135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:45.986322 9135 hybrid_clock.cc:648] HybridClock initialized: now 1790546985986253 us; error 68 us; skew 500 ppm
I20260927 22:09:45.986780 9135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:45 dist-test-slave-mp03 krb5kdc[8616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:45.989837 9135 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:09:45.991384 9135 webserver.cc:460] Webserver started at http://127.0.97.62:37637/ using document root <none> and password file <none>
I20260927 22:09:45.992069 9135 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:45.992231 9135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:45.997659 9135 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 22:09:46.000577 9150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:46.002027 9135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:09:46.002235 9135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "ea5fcd9bf32b468c9c695460cd2ee05b"
format_stamp: "Formatted at 2026-09-27 22:09:43 on dist-test-slave-mp03"
server_key: "7734617e8d9c81931636c2d0961ac44e"
server_key_iv: "ca0a58df7ee2b2e090ea75c26e20a147"
server_key_version: "encryptionkey@0"
I20260927 22:09:46.003331 9135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:46.039330 9135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:46.043723 9135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:46.044028 9135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:46.064030 9135 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44471
I20260927 22:09:46.064049 9201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44471 every 8 connection(s)
I20260927 22:09:46.065655 9135 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:09:46.072319 9202 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:09:46.075286 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 9135
I20260927 22:09:46.075819 9202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b: Bootstrap starting.
I20260927 22:09:46.076231 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41701
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:44471
--webserver_port=43493
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:09:46.096616 9202 log.cc:824] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:09:46.105322 9202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:09:46.105844 9202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b: Bootstrap complete.
I20260927 22:09:46.115054 9202 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5fcd9bf32b468c9c695460cd2ee05b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44471 } }
I20260927 22:09:46.115891 9202 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea5fcd9bf32b468c9c695460cd2ee05b, State: Initialized, Role: FOLLOWER
I20260927 22:09:46.116447 9202 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5fcd9bf32b468c9c695460cd2ee05b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44471 } }
I20260927 22:09:46.116699 9202 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:09:46.116840 9202 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:09:46.117012 9202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:09:46.120702 9202 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5fcd9bf32b468c9c695460cd2ee05b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44471 } }
I20260927 22:09:46.121176 9202 leader_election.cc:302] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea5fcd9bf32b468c9c695460cd2ee05b; no voters:
I20260927 22:09:46.121798 9202 leader_election.cc:288] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:09:46.121924 9206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:09:46.123028 9206 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [term 2 LEADER]: Becoming Leader. State: Replica: ea5fcd9bf32b468c9c695460cd2ee05b, State: Running, Role: LEADER
I20260927 22:09:46.123615 9206 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5fcd9bf32b468c9c695460cd2ee05b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44471 } }
I20260927 22:09:46.123970 9202 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:09:46.126143 9208 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea5fcd9bf32b468c9c695460cd2ee05b. Latest consensus state: current_term: 2 leader_uuid: "ea5fcd9bf32b468c9c695460cd2ee05b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5fcd9bf32b468c9c695460cd2ee05b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44471 } } }
I20260927 22:09:46.126578 9208 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:46.127519 9207 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ea5fcd9bf32b468c9c695460cd2ee05b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5fcd9bf32b468c9c695460cd2ee05b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44471 } } }
I20260927 22:09:46.127951 9207 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b [sys.catalog]: This master's current role is: LEADER
I20260927 22:09:46.135553 9211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:09:46.142480 9211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:09:46.144562 9211 catalog_manager.cc:1259] Loaded cluster ID: 6ef12b037bfa42e59adeff11c66a289e
I20260927 22:09:46.145107 9211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:09:46.150645 9211 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:09:46.152681 9211 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ea5fcd9bf32b468c9c695460cd2ee05b: Loaded TSK: 0
I20260927 22:09:46.154836 9211 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:09:46.439267 9204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:46.439560 9204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:46.439709 9204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:46.447108 9204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:46.447232 9204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:46.447400 9204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:09:46.457381 9204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43493
--tserver_master_addrs=127.0.97.62:44471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:46.458910 9204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:46.460474 9204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:46.475608 9228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:46.476449 9229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:46.476846 9231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:46.478049 9204 server_base.cc:1034] running on GCE node
I20260927 22:09:46.478788 9204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:46.479877 9204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:46.481154 9204 hybrid_clock.cc:648] HybridClock initialized: now 1790546986481090 us; error 57 us; skew 500 ppm
I20260927 22:09:46.481516 9204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:46 dist-test-slave-mp03 krb5kdc[8616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546986, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:46.484804 9204 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 22:09:46.486356 9204 webserver.cc:460] Webserver started at http://127.0.97.1:43493/ using document root <none> and password file <none>
I20260927 22:09:46.487190 9204 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:46.487334 9204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:46.493317 9204 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:09:46.496134 9238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:46.497493 9204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:09:46.497663 9204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "191deb2a46de44248184307a6a68ebc0"
format_stamp: "Formatted at 2026-09-27 22:09:43 on dist-test-slave-mp03"
server_key: "a8cc9622622ce49aca0cd358c4760b8f"
server_key_iv: "7bff1964afebd6aa187353666660499f"
server_key_version: "encryptionkey@0"
I20260927 22:09:46.498638 9204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:46.529608 9204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:46.534144 9204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:46.534433 9204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:46.535432 9204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:46.536769 9204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:46.536892 9204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:46.537045 9204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:46.537134 9204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:46.570686 9204 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41701
I20260927 22:09:46.570793 9350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41701 every 8 connection(s)
I20260927 22:09:46.572121 9204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:09:46.581189 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 9204
I20260927 22:09:46.581897 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:46503
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:44471
--webserver_port=32929
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:09:46 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:46 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:09:46.592244 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:46.575040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46695 (local address 127.0.97.62:44471)
0927 22:09:46.575823 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:46.575839 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:46.576143 (+ 304us) server_negotiation.cc:407] Connection header received
0927 22:09:46.578340 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:46.578352 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:46.578737 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:46.579082 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:46.580564 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:46.582525 (+ 1961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:46.584314 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:46.585429 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:46.585637 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:46.591267 (+ 5630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46695: BlockingRecv error: recv got EOF from 127.0.97.1:46695 (error 108)
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":247,"threads_started":1}
I20260927 22:09:46.592236 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:46.575153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.1:46695)
0927 22:09:46.575956 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:46.575997 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:46.577824 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:46.579226 (+ 1402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:46.579237 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:46.579833 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:46.580363 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:46.580380 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:46.582758 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:46.582763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:46.584157 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:46.584171 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:46.585686 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:46.585692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:46.585886 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:46.586736 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:46.586764 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:46.590978 (+ 4214us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":380,"threads_started":1}
W20260927 22:09:46.593019 9351 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44471 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:44471: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 27 22:09:46 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:46 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:46.602483 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:46.593631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.1:55501)
0927 22:09:46.593806 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:46.593824 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:46.593932 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:46.594318 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:46.594321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:46.594682 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:46.594821 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:46.594829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:46.596379 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:46.596384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:46.597381 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:46.597394 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:46.598497 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:46.598502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:46.598897 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:46.599571 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:46.599588 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:46.602248 (+ 2660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 22:09:46.602829 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:46.593806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55501 (local address 127.0.97.62:44471)
0927 22:09:46.594009 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:46.594017 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:46.594038 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:09:46.594096 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:46.594104 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:46.594188 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:46.594291 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:46.594965 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:46.596201 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:46.597528 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:46.598335 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:46.598482 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:46.602528 (+ 4046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55501: BlockingRecv error: recv got EOF from 127.0.97.1:55501 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 22:09:46 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:46 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:46.611701 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:46.603304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.1:58079)
0927 22:09:46.603467 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:46.603495 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:46.603632 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:46.603974 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:46.603977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:46.604261 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:46.604395 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:46.604402 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:46.605921 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:46.605925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:46.606947 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:46.606960 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:46.607692 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:46.607698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:46.607870 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:46.608509 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:46.608527 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:46.611384 (+ 2857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 22:09:46.611876 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:46.603377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58079 (local address 127.0.97.62:44471)
0927 22:09:46.603576 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:46.603584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:46.603606 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:09:46.603726 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:46.603734 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:46.603833 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:46.603961 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:46.604529 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:46.605767 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:46.607124 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:46.607537 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:46.607676 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:46.611615 (+ 3939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58079: BlockingRecv error: recv got EOF from 127.0.97.1:58079 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 22:09:46.612058 9351 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:09:46.910944 9355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:46.911286 9355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:46.911458 9355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:46.918881 9355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:46.919049 9355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:46.919224 9355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:09:46.929287 9355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:46503
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=32929
--tserver_master_addrs=127.0.97.62:44471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:46.930740 9355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:46.932140 9355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:46.945976 9360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:46.945930 9363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:46.946009 9361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:46.946875 9355 server_base.cc:1034] running on GCE node
I20260927 22:09:46.947479 9355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:46.948410 9355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:46.949631 9355 hybrid_clock.cc:648] HybridClock initialized: now 1790546986949585 us; error 63 us; skew 500 ppm
I20260927 22:09:46.949954 9355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:46 dist-test-slave-mp03 krb5kdc[8616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546986, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:46.953086 9355 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 22:09:46.954430 9355 webserver.cc:460] Webserver started at http://127.0.97.2:32929/ using document root <none> and password file <none>
I20260927 22:09:46.955171 9355 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:46.955282 9355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:46.960681 9355 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:09:46.963356 9370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:46.964736 9355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:09:46.964918 9355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "eafe04d23a224b55a5a634d29e59669f"
format_stamp: "Formatted at 2026-09-27 22:09:44 on dist-test-slave-mp03"
server_key: "6162766aa132f2889640268b87341669"
server_key_iv: "c527fc730d6c2c941c15336969e28488"
server_key_version: "encryptionkey@0"
I20260927 22:09:46.965855 9355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:46.986744 9355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:46.990943 9355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:46.991210 9355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:46.992161 9355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:46.993495 9355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:46.993620 9355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:46.993742 9355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:46.993832 9355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:47.027367 9355 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46503
I20260927 22:09:47.027455 9482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46503 every 8 connection(s)
I20260927 22:09:47.028858 9355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:09:47.037369 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 9355
I20260927 22:09:47.038108 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36523
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:44471
--webserver_port=41897
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:09:47 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:47 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:09:47.046151 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:47.031565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51315 (local address 127.0.97.62:44471)
0927 22:09:47.031768 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:47.031774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:47.033009 (+ 1235us) server_negotiation.cc:407] Connection header received
0927 22:09:47.034849 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:47.034854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:47.034918 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:47.035006 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:47.036497 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.037170 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.039793 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.040242 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.040349 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:47.045913 (+ 5564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51315: BlockingRecv error: recv got EOF from 127.0.97.2:51315 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:09:47.046960 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:47.032119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.2:51315)
0927 22:09:47.032850 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:47.032889 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:47.034527 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:47.035216 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:47.035225 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:47.035793 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:47.036346 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:47.036363 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.037332 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.037341 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:47.039599 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:47.039617 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.040483 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.040493 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:47.040837 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:47.041589 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:47.041627 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:47.045714 (+ 4087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":296,"threads_started":1}
W20260927 22:09:47.047643 9483 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44471 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:44471: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 27 22:09:47 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:47 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:47.056202 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:47.048260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.2:44153)
0927 22:09:47.048434 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:47.048454 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:47.048559 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:47.048904 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:47.048909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:47.049260 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:47.049385 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:47.049393 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.050342 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.050347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:47.051305 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:47.051315 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.051965 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.051970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:47.052137 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:47.052808 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:47.052824 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:47.055911 (+ 3087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 22:09:47.056352 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:47.048340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44153 (local address 127.0.97.62:44471)
0927 22:09:47.048506 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:47.048512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:47.048530 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:09:47.048715 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:47.048718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:47.048787 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:47.048881 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:47.049519 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.050211 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.051438 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.051824 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.051955 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:47.056114 (+ 4159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44153: BlockingRecv error: recv got EOF from 127.0.97.2:44153 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 22:09:47 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:47 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:47.066231 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:47.057061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.2:51985)
0927 22:09:47.057291 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:47.057313 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:47.057422 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:47.057735 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:47.057743 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:47.058163 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:47.058389 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:47.058403 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.059428 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.059433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:47.061233 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:47.061249 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.061922 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.061928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:47.062100 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:47.062656 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:47.062678 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:47.065919 (+ 3241us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88}
W20260927 22:09:47.066332 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:47.057208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51985 (local address 127.0.97.62:44471)
0927 22:09:47.057351 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:47.057355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:47.057371 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:09:47.057547 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:47.057550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:47.057602 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:47.057670 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:47.058519 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.059298 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.061400 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.061762 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.061880 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:47.066124 (+ 4244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51985: BlockingRecv error: recv got EOF from 127.0.97.2:51985 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 22:09:47.066749 9483 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:09:47.369772 9486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:09:47.370082 9486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:09:47.370241 9486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:09:47.378517 9486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:09:47.378695 9486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:09:47.378882 9486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:09:47.388713 9486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=41897
--tserver_master_addrs=127.0.97.62:44471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:09:47.390125 9486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:09:47.391480 9486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:09:47.404481 9491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:47.404507 9492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:09:47.404639 9494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:09:47.405251 9486 server_base.cc:1034] running on GCE node
I20260927 22:09:47.405833 9486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:09:47.406849 9486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:09:47.408075 9486 hybrid_clock.cc:648] HybridClock initialized: now 1790546987408022 us; error 62 us; skew 500 ppm
I20260927 22:09:47.408403 9486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:09:47 dist-test-slave-mp03 krb5kdc[8616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546987, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:09:47.411464 9486 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 22:09:47.412839 9486 webserver.cc:460] Webserver started at http://127.0.97.3:41897/ using document root <none> and password file <none>
I20260927 22:09:47.413568 9486 fs_manager.cc:359] Metadata directory not provided
I20260927 22:09:47.413722 9486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:09:47.419015 9486 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:09:47.421641 9501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:47.422999 9486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:09:47.423211 9486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "6e982a0d66b94fd2bd0ffefb5c930cfd"
format_stamp: "Formatted at 2026-09-27 22:09:44 on dist-test-slave-mp03"
server_key: "8f0033be597a7856f5fba572bfda686b"
server_key_iv: "e25602adce3c013f5581806d6a1e6de5"
server_key_version: "encryptionkey@0"
I20260927 22:09:47.424153 9486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:09:47.451429 9486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:09:47.455659 9486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:09:47.455950 9486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:09:47.456933 9486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:09:47.458249 9486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:09:47.458389 9486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:47.458528 9486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:09:47.458637 9486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:09:47.490983 9486 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36523
I20260927 22:09:47.491094 9613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36523 every 8 connection(s)
I20260927 22:09:47.492412 9486 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:09:47.501609 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 9486
Sep 27 22:09:47 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:47 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:09:47.508658 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:47.494886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45889 (local address 127.0.97.62:44471)
0927 22:09:47.495053 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:47.495058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:47.496279 (+ 1221us) server_negotiation.cc:407] Connection header received
0927 22:09:47.498085 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:47.498089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:47.498158 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:47.498248 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:47.499749 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.500455 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.502037 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.502586 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.502760 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:47.508458 (+ 5698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45889: BlockingRecv error: recv got EOF from 127.0.97.3:45889 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:09:47.509603 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:47.495407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.3:45889)
0927 22:09:47.496101 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:47.496145 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:47.497795 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:47.498453 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:47.498461 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:47.499033 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:47.499593 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:47.499609 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.500592 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.500596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:47.501893 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:47.501905 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.502817 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.502825 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:47.503104 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:47.503953 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:47.503981 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:47.508327 (+ 4346us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":276,"threads_started":1}
W20260927 22:09:47.510214 9614 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44471 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:44471: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 27 22:09:47 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:47 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:47.518872 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:47.510803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.3:55061)
0927 22:09:47.510969 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:47.510990 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:47.511096 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:47.511779 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:47.511785 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:47.512190 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:47.512382 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:47.512394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.513370 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.513375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:47.514304 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:47.514315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.515077 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.515083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:47.515277 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:47.515794 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:47.515810 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:47.518542 (+ 2732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 22:09:47.518998 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:47.511125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55061 (local address 127.0.97.62:44471)
0927 22:09:47.511528 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:47.511532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:47.511545 (+ 13us) server_negotiation.cc:407] Connection header received
0927 22:09:47.511586 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:47.511589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:47.511637 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:47.511709 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:47.512537 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.513236 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.514450 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.514925 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.515071 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:47.518753 (+ 3682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55061: BlockingRecv error: recv got EOF from 127.0.97.3:55061 (error 108)
Metrics: {"server-negotiator.queue_time_us":308}
I20260927 22:09:47.520572 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:47.502793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60552 (local address 127.0.97.62:44471)
0927 22:09:47.504382 (+ 1589us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:47.504385 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:47.504398 (+ 13us) server_negotiation.cc:407] Connection header received
0927 22:09:47.504431 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:47.504435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:47.504481 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:47.504567 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:47.506381 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.507418 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.509280 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.510455 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.510648 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:47.513648 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:09:47.513655 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:09:47.513672 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:09:47.513706 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:09:47.517027 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:47.517698 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:47.517705 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:47.517712 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:47.517880 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:09:47.518362 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:09:47.518369 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:09:47.518372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:09:47.518893 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:09:47.519226 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0927 22:09:47.519903 (+ 677us) server_negotiation.cc:299] Negotiation successful
0927 22:09:47.520290 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1502,"thread_start_us":803,"threads_started":1}
Sep 27 22:09:47 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:47 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:47.527550 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:47.519714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.3:53249)
0927 22:09:47.519945 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:47.519965 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:47.520058 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:47.520385 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:47.520389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:47.520723 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:47.520884 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:47.520896 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.522074 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.522078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:47.523160 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:47.523173 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.523862 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.523868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:47.524031 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:47.524623 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:47.524672 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:47.527230 (+ 2558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126}
W20260927 22:09:47.527673 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:47.519898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53249 (local address 127.0.97.62:44471)
0927 22:09:47.520158 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:47.520163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:47.520184 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:09:47.520217 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:47.520220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:47.520265 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:47.520371 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:47.521030 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.521936 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.523281 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.523726 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.523853 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:47.527403 (+ 3550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53249: BlockingRecv error: recv got EOF from 127.0.97.3:53249 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
W20260927 22:09:47.527942 9614 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 22:09:47 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:47 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:47.621443 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:47.612635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.1:60501)
0927 22:09:47.613149 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:47.613167 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:47.613320 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:47.613673 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:47.613678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:47.614181 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:47.614365 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:47.614374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.615428 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.615440 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:47.616429 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:47.616440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.617178 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.617185 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:47.617353 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:47.617999 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:47.618024 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:47.621126 (+ 3102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":258,"threads_started":1}
W20260927 22:09:47.621632 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:47.612726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60501 (local address 127.0.97.62:44471)
0927 22:09:47.612970 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:47.612978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:47.613247 (+ 269us) server_negotiation.cc:407] Connection header received
0927 22:09:47.613461 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:47.613465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:47.613544 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:47.613648 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:47.614504 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.615283 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.616572 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:47.616994 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:47.617138 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:47.621366 (+ 4228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60501: BlockingRecv error: recv got EOF from 127.0.97.1:60501 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 22:09:48 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:48 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:48.076287 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:48.067398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.2:40455)
0927 22:09:48.067933 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:48.067950 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:48.068060 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:48.068389 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:48.068392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:48.068939 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:48.069241 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:48.069255 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:48.070630 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:48.070635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:48.071803 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:48.071814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:48.072497 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:48.072502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:48.072627 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:48.073198 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:48.073219 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:48.075957 (+ 2738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":294,"threads_started":1}
W20260927 22:09:48.076375 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:48.067464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40455 (local address 127.0.97.62:44471)
0927 22:09:48.067671 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:48.067678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:48.068029 (+ 351us) server_negotiation.cc:407] Connection header received
0927 22:09:48.068165 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:48.068170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:48.068247 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:48.068351 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:48.069396 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:48.070456 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:48.071934 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:48.072341 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:48.072435 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:48.076138 (+ 3703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40455: BlockingRecv error: recv got EOF from 127.0.97.2:40455 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 22:09:48 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:48 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:48.546631 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:48.528640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.3:56575)
0927 22:09:48.529195 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:48.529210 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:48.529368 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:48.529738 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:48.529742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:48.530207 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:48.530388 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:48.530396 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:48.531495 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:48.531500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:48.532582 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:48.532594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:48.533494 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:48.533501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:48.533698 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:48.542572 (+ 8874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:48.542636 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:48.546323 (+ 3687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":343,"threads_started":1}
W20260927 22:09:48.546824 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:48.528705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56575 (local address 127.0.97.62:44471)
0927 22:09:48.529023 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:48.529033 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:48.529302 (+ 269us) server_negotiation.cc:407] Connection header received
0927 22:09:48.529538 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:48.529542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:48.529601 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:48.529708 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:48.530547 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:48.531343 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:48.532730 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:48.533292 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:48.533421 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:48.546533 (+ 13112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56575: BlockingRecv error: recv got EOF from 127.0.97.3:56575 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 22:09:48 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:48 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:48.631338 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:48.622381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.1:46317)
0927 22:09:48.622943 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:48.622967 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:48.623078 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:48.623411 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:48.623414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:48.623779 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:48.624110 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:48.624130 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:48.625155 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:48.625162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:48.626090 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:48.626101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:48.626862 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:48.626868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:48.627043 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:48.627898 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:48.627920 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:48.631035 (+ 3115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":346,"threads_started":1}
W20260927 22:09:48.631621 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:48.622451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46317 (local address 127.0.97.62:44471)
0927 22:09:48.622657 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:48.622663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:48.623036 (+ 373us) server_negotiation.cc:407] Connection header received
0927 22:09:48.623191 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:48.623194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:48.623275 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:48.623359 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:48.624257 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:48.625023 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:48.626229 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:48.626714 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:48.626849 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:48.631227 (+ 4378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46317: BlockingRecv error: recv got EOF from 127.0.97.1:46317 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 22:09:49 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:49 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:49.090402 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:49.077500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.2:45033)
0927 22:09:49.078243 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:49.078260 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:49.078423 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:49.078875 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:49.078882 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:49.080038 (+ 1156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:49.080261 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:49.080275 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:49.081780 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:49.081787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:49.083629 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:49.083645 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:49.084918 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:49.084925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:49.085083 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:49.085757 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:49.085774 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:49.089920 (+ 4146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":512,"threads_started":1}
W20260927 22:09:49.090494 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:49.077549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45033 (local address 127.0.97.62:44471)
0927 22:09:49.077773 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:49.077780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:49.078332 (+ 552us) server_negotiation.cc:407] Connection header received
0927 22:09:49.078571 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:49.078579 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:49.078700 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:49.078832 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:49.080453 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:49.081637 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:49.083823 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:49.084641 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:49.084764 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:49.090187 (+ 5423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45033: BlockingRecv error: recv got EOF from 127.0.97.2:45033 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 22:09:49 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:49 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:49.556228 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:49.547652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.3:47779)
0927 22:09:49.548202 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:49.548218 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:49.548372 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:49.548738 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:49.548743 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:49.549161 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:49.549297 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:49.549304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:49.550334 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:49.550341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:49.551404 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:49.551415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:49.552254 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:49.552259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:49.552440 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:49.553148 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:49.553166 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:49.556002 (+ 2836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":307,"threads_started":1}
W20260927 22:09:49.556404 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:49.547872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47779 (local address 127.0.97.62:44471)
0927 22:09:49.548088 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:49.548095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:49.548301 (+ 206us) server_negotiation.cc:407] Connection header received
0927 22:09:49.548501 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:49.548506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:49.548578 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:49.548679 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:49.549445 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:49.550193 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:49.551532 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:49.552051 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:49.552209 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:49.556177 (+ 3968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47779: BlockingRecv error: recv got EOF from 127.0.97.3:47779 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 22:09:49 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:49 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:49.640851 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:49.632394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.1:34717)
0927 22:09:49.632916 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:49.632933 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:49.633052 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:49.633357 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:49.633360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:49.633719 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:49.633895 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:49.633903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:49.635014 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:49.635019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:49.635961 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:49.635973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:49.636887 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:49.636893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:49.637052 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:49.637793 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:49.637820 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:49.640460 (+ 2640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":295,"threads_started":1}
W20260927 22:09:49.640894 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:49.632461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34717 (local address 127.0.97.62:44471)
0927 22:09:49.632669 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:49.632673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:49.633010 (+ 337us) server_negotiation.cc:407] Connection header received
0927 22:09:49.633160 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:49.633163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:49.633225 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:49.633313 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:49.634053 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:49.634869 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:49.636123 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:49.636685 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:49.636827 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:49.640646 (+ 3819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:34717: BlockingRecv error: recv got EOF from 127.0.97.1:34717 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 22:09:50 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:50 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:50.099923 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:50.091314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.2:53503)
0927 22:09:50.091837 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:50.091860 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:50.091970 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:50.092329 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:50.092332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:50.092663 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:50.092822 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:50.092829 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:50.093867 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:50.093871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:50.094919 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:50.094937 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:50.095877 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:50.095902 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:50.096090 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:50.096785 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:50.096803 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:50.099664 (+ 2861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":284,"threads_started":1}
W20260927 22:09:50.100267 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:50.091473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53503 (local address 127.0.97.62:44471)
0927 22:09:50.091645 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:50.091650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:50.091934 (+ 284us) server_negotiation.cc:407] Connection header received
0927 22:09:50.092111 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:50.092116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:50.092183 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:50.092288 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:50.092979 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:50.093752 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:50.095089 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:50.095687 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:50.095862 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:50.099917 (+ 4055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53503: BlockingRecv error: recv got EOF from 127.0.97.2:53503 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 22:09:50 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:50 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:50.566233 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:50.557060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.3:54423)
0927 22:09:50.557585 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:50.557600 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:50.557757 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:50.558174 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:50.558179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:50.558753 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:50.558913 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:50.558922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:50.559998 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:50.560003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:50.560892 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:50.560903 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:50.561588 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:50.561594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:50.561775 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:50.562522 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:50.562573 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:50.565971 (+ 3398us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":256,"threads_started":1}
W20260927 22:09:50.566416 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:50.557302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54423 (local address 127.0.97.62:44471)
0927 22:09:50.557504 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:50.557511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:50.557681 (+ 170us) server_negotiation.cc:407] Connection header received
0927 22:09:50.557887 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:50.557893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:50.557969 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:50.558122 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:50.559072 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:50.559865 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:50.561042 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:50.561440 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:50.561573 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:50.566189 (+ 4616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54423: BlockingRecv error: recv got EOF from 127.0.97.3:54423 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 22:09:50 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:50 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:09:50.650265 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:50.641960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37799 (local address 127.0.97.62:44471)
0927 22:09:50.642226 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:50.642232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:50.642527 (+ 295us) server_negotiation.cc:407] Connection header received
0927 22:09:50.642790 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:50.642794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:50.642855 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:50.642935 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:50.643584 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:50.644288 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:50.645474 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:50.645885 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:50.645979 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:50.650067 (+ 4088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37799: BlockingRecv error: recv got EOF from 127.0.97.1:37799 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
I20260927 22:09:50.650346 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:50.641906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.1:37799)
0927 22:09:50.642397 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:50.642413 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:50.642523 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:50.642977 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:50.642979 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:50.643318 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:50.643465 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:50.643473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:50.644411 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:50.644415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:50.645347 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:50.645357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:50.646006 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:50.646010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:50.646144 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:50.646810 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:50.646828 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:50.649926 (+ 3098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":288,"threads_started":1}
Sep 27 22:09:51 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:51 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:51.109748 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:51.100978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.2:53423)
0927 22:09:51.101452 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:51.101468 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:51.101590 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:51.102075 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:51.102079 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:51.102430 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:51.102576 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:51.102584 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:51.103860 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:51.103864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:51.104867 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:51.104878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:51.105559 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:51.105564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:51.105696 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:51.106373 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:51.106390 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:51.109421 (+ 3031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":257,"threads_started":1}
W20260927 22:09:51.109822 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:51.101043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53423 (local address 127.0.97.62:44471)
0927 22:09:51.101298 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:51.101305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:51.101604 (+ 299us) server_negotiation.cc:407] Connection header received
0927 22:09:51.101685 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:51.101690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:51.101770 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:51.101884 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:51.102766 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:51.103726 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:51.104988 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:51.105412 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:51.105506 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:51.109605 (+ 4099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53423: BlockingRecv error: recv got EOF from 127.0.97.2:53423 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 22:09:51 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:51 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:51.576700 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:51.567197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.3:44565)
0927 22:09:51.567702 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:51.567718 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:51.567893 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:51.568383 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:51.568390 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:51.568887 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:51.569042 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:51.569049 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:51.570495 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:51.570500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:51.571535 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:51.571546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:51.572417 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:51.572421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:51.572548 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:51.573225 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:51.573255 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:51.576368 (+ 3113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":269,"threads_started":1}
W20260927 22:09:51.576882 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:51.567271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44565 (local address 127.0.97.62:44471)
0927 22:09:51.567468 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:51.567476 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:51.567810 (+ 334us) server_negotiation.cc:407] Connection header received
0927 22:09:51.568095 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:51.568103 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:51.568197 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:51.568325 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:51.569182 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:51.570340 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:51.571673 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:51.572274 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:51.572401 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:51.576627 (+ 4226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44565: BlockingRecv error: recv got EOF from 127.0.97.3:44565 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 22:09:51 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:51 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:51.660151 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:51.651318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.1:36291)
0927 22:09:51.651990 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:51.652009 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:51.652126 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:51.652427 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:51.652430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:51.652854 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:51.653016 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:51.653024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:51.654280 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:51.654287 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:51.655287 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:51.655298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:51.655997 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:51.656003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:51.656175 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:51.656821 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:51.656840 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:51.659798 (+ 2958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":410,"threads_started":1}
W20260927 22:09:51.660254 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:51.651518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36291 (local address 127.0.97.62:44471)
0927 22:09:51.651692 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:51.651696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:51.652084 (+ 388us) server_negotiation.cc:407] Connection header received
0927 22:09:51.652226 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:51.652230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:51.652302 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:51.652385 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:51.653189 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:51.654131 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:51.655426 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:51.655834 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:51.655963 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:51.660001 (+ 4038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36291: BlockingRecv error: recv got EOF from 127.0.97.1:36291 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 22:09:52 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:52 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:52.119233 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:52.110776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.2:52507)
0927 22:09:52.111449 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:52.111465 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:52.111648 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:52.111989 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:52.111994 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:52.112395 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:52.112550 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:52.112558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:52.113568 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:52.113577 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:52.114475 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:52.114485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:52.115245 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:52.115250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:52.115422 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:52.116022 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:52.116039 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:52.118979 (+ 2940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":356,"threads_started":1}
W20260927 22:09:52.119464 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:52.111007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52507 (local address 127.0.97.62:44471)
0927 22:09:52.111260 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:52.111266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:52.111585 (+ 319us) server_negotiation.cc:407] Connection header received
0927 22:09:52.111791 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:52.111795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:52.111855 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:52.111960 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:52.112693 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:52.113429 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:52.114633 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:52.115090 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:52.115230 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:52.119161 (+ 3931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52507: BlockingRecv error: recv got EOF from 127.0.97.2:52507 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 27 22:09:52 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:52 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:52.586623 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:52.577862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.3:57261)
0927 22:09:52.578509 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:52.578526 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:52.578705 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:52.579010 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:52.579013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:52.579377 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:52.579537 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:52.579545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:52.580534 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:52.580539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:52.581664 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:52.581677 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:52.582346 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:52.582350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:52.582478 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:52.583116 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:52.583135 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:52.586249 (+ 3114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":399,"threads_started":1}
W20260927 22:09:52.586715 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:52.577918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57261 (local address 127.0.97.62:44471)
0927 22:09:52.578137 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:52.578147 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:52.578652 (+ 505us) server_negotiation.cc:407] Connection header received
0927 22:09:52.578818 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:52.578822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:52.578884 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:52.578966 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:52.579656 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:52.580372 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:52.581797 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:52.582207 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:52.582298 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:52.586445 (+ 4147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57261: BlockingRecv error: recv got EOF from 127.0.97.3:57261 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 22:09:52 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:52 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:52.669625 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:52.661165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.1:40823)
0927 22:09:52.661767 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:52.661790 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:52.661936 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:52.662306 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:52.662310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:52.662866 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:52.663012 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:52.663026 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:52.664019 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:52.664025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:52.664932 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:52.664943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:52.665665 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:52.665671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:52.665841 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:52.666446 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:52.666464 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:52.669396 (+ 2932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":343,"threads_started":1}
W20260927 22:09:52.669800 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:52.661273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40823 (local address 127.0.97.62:44471)
0927 22:09:52.661507 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:52.661516 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:52.661883 (+ 367us) server_negotiation.cc:407] Connection header received
0927 22:09:52.662086 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:52.662097 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:52.662170 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:52.662275 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:52.663156 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:52.663881 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:52.665074 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:52.665512 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:52.665662 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:52.669538 (+ 3876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40823: BlockingRecv error: recv got EOF from 127.0.97.1:40823 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 22:09:53 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:53 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:53.128620 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:53.120243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.2:48751)
0927 22:09:53.120837 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:53.120855 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:53.120999 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:53.121310 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:53.121313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:53.121701 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:53.121878 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:53.121888 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:53.122910 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:53.122915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:53.124040 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:53.124053 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:53.124717 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:53.124727 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:53.124931 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:53.125523 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:53.125541 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:53.128293 (+ 2752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":371,"threads_started":1}
W20260927 22:09:53.128830 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:53.120298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48751 (local address 127.0.97.62:44471)
0927 22:09:53.120527 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:53.120536 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:53.120945 (+ 409us) server_negotiation.cc:407] Connection header received
0927 22:09:53.121112 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:53.121116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:53.121175 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:53.121247 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:53.122042 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:53.122768 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:53.124181 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:53.124560 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:53.124648 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:53.128593 (+ 3945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48751: BlockingRecv error: recv got EOF from 127.0.97.2:48751 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 22:09:53 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:53 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:53.601830 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:53.587555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.3:36635)
0927 22:09:53.588101 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:53.588117 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:53.588231 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:53.588514 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:53.588517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:53.588917 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:53.589070 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:53.589078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:53.590013 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:53.590017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:53.590989 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:53.591000 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:53.591697 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:53.591701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:53.591817 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:53.598519 (+ 6702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:53.598545 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:53.601502 (+ 2957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":318,"threads_started":1}
W20260927 22:09:53.602012 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:53.587615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36635 (local address 127.0.97.62:44471)
0927 22:09:53.587774 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:53.587778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:53.588194 (+ 416us) server_negotiation.cc:407] Connection header received
0927 22:09:53.588329 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:53.588332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:53.588389 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:53.588468 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:53.589185 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:53.589894 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:53.591110 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:53.591524 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:53.591621 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:53.601762 (+ 10141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36635: BlockingRecv error: recv got EOF from 127.0.97.3:36635 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 22:09:53 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:53 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:53.679266 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:53.670872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.1:49017)
0927 22:09:53.671512 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:53.671532 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:53.671688 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:53.672094 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:53.672097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:53.672614 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:53.672783 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:53.672792 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:53.673905 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:53.673909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:53.674812 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:53.674822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:53.675526 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:53.675530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:53.675632 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:53.676229 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:53.676245 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:53.679034 (+ 2789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":263,"threads_started":1}
W20260927 22:09:53.679454 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:53.670914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49017 (local address 127.0.97.62:44471)
0927 22:09:53.671069 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:53.671073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:53.671673 (+ 600us) server_negotiation.cc:407] Connection header received
0927 22:09:53.671806 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:53.671814 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:53.671925 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:53.672058 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:53.672920 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:53.673772 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:53.674963 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:53.675401 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:53.675501 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:53.679245 (+ 3744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49017: BlockingRecv error: recv got EOF from 127.0.97.1:49017 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 22:09:54 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:54 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:54.138401 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:54.129709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.2:52577)
0927 22:09:54.130292 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:54.130307 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:54.130472 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:54.130818 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:54.130822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:54.131238 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:54.131409 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:54.131417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:54.132419 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:54.132425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:54.133322 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:54.133332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:54.134016 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:54.134022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:54.134181 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:54.134945 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:54.134967 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:54.138130 (+ 3163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":251,"threads_started":1}
W20260927 22:09:54.138540 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:54.129762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52577 (local address 127.0.97.62:44471)
0927 22:09:54.130048 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:54.130056 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:54.130402 (+ 346us) server_negotiation.cc:407] Connection header received
0927 22:09:54.130623 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:54.130628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:54.130688 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:54.130792 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:54.131568 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:54.132288 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:54.133453 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:54.133858 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:54.134003 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:54.138311 (+ 4308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52577: BlockingRecv error: recv got EOF from 127.0.97.2:52577 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Sep 27 22:09:54 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:54 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:54.611224 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:54.602801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.3:59999)
0927 22:09:54.603494 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:54.603513 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:54.603658 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:54.604009 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:54.604014 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:54.604416 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:54.604574 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:54.604582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:54.605591 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:54.605597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:54.606535 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:54.606546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:54.607352 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:54.607358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:54.607520 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:54.608124 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:54.608141 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:54.610851 (+ 2710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":315,"threads_started":1}
W20260927 22:09:54.611253 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:54.602894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59999 (local address 127.0.97.62:44471)
0927 22:09:54.603133 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:54.603141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:54.603580 (+ 439us) server_negotiation.cc:407] Connection header received
0927 22:09:54.603805 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:54.603810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:54.603883 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:54.603983 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:54.604717 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:54.605460 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:54.606717 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:54.607200 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:54.607337 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:54.611045 (+ 3708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59999: BlockingRecv error: recv got EOF from 127.0.97.3:59999 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 27 22:09:54 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:54 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:54.688417 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:54.680134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.1:49031)
0927 22:09:54.680636 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:54.680659 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:54.680815 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:54.681203 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:54.681208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:54.681632 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:54.681775 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:54.681783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:54.682849 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:54.682855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:54.683798 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:54.683809 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:54.684697 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:54.684702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:54.684838 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:54.685404 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:54.685421 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:54.688096 (+ 2675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":247,"threads_started":1}
W20260927 22:09:54.688604 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:54.680369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49031 (local address 127.0.97.62:44471)
0927 22:09:54.680596 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:54.680603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:54.680740 (+ 137us) server_negotiation.cc:407] Connection header received
0927 22:09:54.680945 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:54.680950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:54.681054 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:54.681169 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:54.681917 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:54.682720 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:54.684014 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:54.684514 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:54.684617 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:54.688346 (+ 3729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49031: BlockingRecv error: recv got EOF from 127.0.97.1:49031 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 27 22:09:55 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:55 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:55.148175 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:55.139722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.2:57005)
0927 22:09:55.140271 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:55.140286 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:55.140395 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:55.140694 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:55.140697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:55.141074 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:55.141212 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:55.141220 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:55.142420 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:55.142424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:55.143506 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:55.143517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:55.144227 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:55.144232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:55.144350 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:55.145014 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:55.145030 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:55.147910 (+ 2880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":271,"threads_started":1}
W20260927 22:09:55.148484 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:55.139774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57005 (local address 127.0.97.62:44471)
0927 22:09:55.140005 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:55.140016 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:55.140374 (+ 358us) server_negotiation.cc:407] Connection header received
0927 22:09:55.140497 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:55.140501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:55.140565 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:55.140656 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:55.141364 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:55.142264 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:55.143648 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:55.144076 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:55.144179 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:55.148154 (+ 3975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57005: BlockingRecv error: recv got EOF from 127.0.97.2:57005 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 22:09:55 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:55 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:55.621341 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:55.612266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.3:49931)
0927 22:09:55.612712 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:55.612727 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:55.612867 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:55.613171 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:55.613176 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:55.613721 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:55.613961 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:55.613976 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:55.615399 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:55.615404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:55.616409 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:55.616423 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:55.617169 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:55.617178 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:55.617337 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:55.617982 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:55.618002 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:55.621017 (+ 3015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":268,"threads_started":1}
W20260927 22:09:55.621498 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:55.612314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49931 (local address 127.0.97.62:44471)
0927 22:09:55.612552 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:55.612560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:55.612834 (+ 274us) server_negotiation.cc:407] Connection header received
0927 22:09:55.612969 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:55.612973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:55.613034 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:55.613113 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:55.614111 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:55.615232 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:55.616546 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:55.616947 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:55.617042 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:55.621229 (+ 4187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49931: BlockingRecv error: recv got EOF from 127.0.97.3:49931 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 22:09:55 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:55 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:55.697856 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:55.689458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.1:35753)
0927 22:09:55.689983 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:55.689999 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:55.690145 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:55.690527 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:55.690532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:55.690939 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:55.691100 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:55.691108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:55.692109 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:55.692116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:55.693060 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:55.693071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:55.693781 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:55.693787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:55.693967 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:55.694529 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:55.694547 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:55.697497 (+ 2950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":266,"threads_started":1}
W20260927 22:09:55.697988 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:55.689508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35753 (local address 127.0.97.62:44471)
0927 22:09:55.689780 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:55.689788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:55.690083 (+ 295us) server_negotiation.cc:407] Connection header received
0927 22:09:55.690305 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:55.690310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:55.690391 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:55.690494 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:55.691244 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:55.691979 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:55.693198 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:55.693625 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:55.693769 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:55.697752 (+ 3983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35753: BlockingRecv error: recv got EOF from 127.0.97.1:35753 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 22:09:56 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:56 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:56.158046 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:56.149305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.2:58461)
0927 22:09:56.149802 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:56.149819 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:56.149963 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:56.150294 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:56.150297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:56.150781 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:56.150992 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:56.151007 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:56.152265 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:56.152272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:56.153283 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:56.153294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:56.154118 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:56.154122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:56.154235 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:56.154910 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:56.154926 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:56.157649 (+ 2723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":256,"threads_started":1}
W20260927 22:09:56.158169 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:56.149445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58461 (local address 127.0.97.62:44471)
0927 22:09:56.149686 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:56.149692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:56.149931 (+ 239us) server_negotiation.cc:407] Connection header received
0927 22:09:56.150077 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:56.150081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:56.150145 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:56.150280 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:56.151152 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:56.152118 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:56.153498 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:56.153988 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:56.154097 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:56.157898 (+ 3801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58461: BlockingRecv error: recv got EOF from 127.0.97.2:58461 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 27 22:09:56 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:56 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:56.631512 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:56.622487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.3:43845)
0927 22:09:56.623029 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:56.623044 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:56.623214 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:56.623569 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:56.623573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:56.623990 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:56.624149 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:56.624160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:56.625368 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:56.625376 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:56.626361 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:56.626373 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:56.627159 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:56.627165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:56.627344 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:56.627924 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:56.627942 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:56.631171 (+ 3229us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":296,"threads_started":1}
W20260927 22:09:56.631731 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:56.622629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43845 (local address 127.0.97.62:44471)
0927 22:09:56.622852 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:56.622858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:56.623131 (+ 273us) server_negotiation.cc:407] Connection header received
0927 22:09:56.623347 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:56.623354 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:56.623436 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:56.623543 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:56.624301 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:56.625128 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:56.626505 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:56.626997 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:56.627138 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:56.631479 (+ 4341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43845: BlockingRecv error: recv got EOF from 127.0.97.3:43845 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 22:09:56 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:56 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:56.707547 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:56.698935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.1:53875)
0927 22:09:56.699551 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:56.699566 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:56.699673 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:56.699982 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:56.699986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:56.700348 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:56.700487 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:56.700495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:56.701616 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:56.701622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:56.702853 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:56.702874 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:56.703817 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:56.703822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:56.703939 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:56.704493 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:56.704510 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:56.707253 (+ 2743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":398,"threads_started":1}
W20260927 22:09:56.707751 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:56.699115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53875 (local address 127.0.97.62:44471)
0927 22:09:56.699358 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:56.699367 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:56.699679 (+ 312us) server_negotiation.cc:407] Connection header received
0927 22:09:56.699772 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:56.699776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:56.699851 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:56.699943 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:56.700629 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:56.701469 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:56.703006 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:56.703686 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:56.703778 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:56.707474 (+ 3696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53875: BlockingRecv error: recv got EOF from 127.0.97.1:53875 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 22:09:57 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:57 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:57.167563 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:57.159101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.2:44547)
0927 22:09:57.159688 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:57.159710 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:57.159887 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:57.160255 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:57.160260 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:57.160669 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:57.160805 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:57.160813 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:57.161801 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:57.161807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:57.162828 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:57.162840 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:57.163564 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:57.163571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:57.163752 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:57.164320 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:57.164338 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:57.167257 (+ 2919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":293,"threads_started":1}
W20260927 22:09:57.167755 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:57.159209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44547 (local address 127.0.97.62:44471)
0927 22:09:57.159378 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:57.159383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:57.159816 (+ 433us) server_negotiation.cc:407] Connection header received
0927 22:09:57.160044 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:57.160050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:57.160120 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:57.160223 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:57.160951 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:57.161659 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:57.162965 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:57.163376 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:57.163531 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:57.167473 (+ 3942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44547: BlockingRecv error: recv got EOF from 127.0.97.2:44547 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 22:09:57 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:57 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:57.640252 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:57.632404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.3:51439)
0927 22:09:57.632938 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:57.632954 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:57.633059 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:57.633367 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:57.633370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:57.633749 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:57.633909 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:57.633918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:57.634959 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:57.634963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:57.635858 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:57.635869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:57.636520 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:57.636524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:57.636639 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:57.637274 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:57.637291 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:57.640039 (+ 2748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":309,"threads_started":1}
W20260927 22:09:57.640786 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:57.632565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51439 (local address 127.0.97.62:44471)
0927 22:09:57.632763 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:57.632768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:57.633020 (+ 252us) server_negotiation.cc:407] Connection header received
0927 22:09:57.633155 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:57.633159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:57.633225 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:57.633330 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:57.634064 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:57.634834 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:57.635976 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:57.636393 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:57.636484 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:57.640527 (+ 4043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51439: BlockingRecv error: recv got EOF from 127.0.97.3:51439 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 27 22:09:57 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:57 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:57.716670 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:57.708440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.1:35025)
0927 22:09:57.708960 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:57.708977 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:57.709088 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:57.709397 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:57.709400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:57.709803 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:57.709970 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:57.709978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:57.711078 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:57.711082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:57.712004 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:57.712015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:57.712719 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:57.712723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:57.712843 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:57.713493 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:57.713510 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:57.716441 (+ 2931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":296,"threads_started":1}
W20260927 22:09:57.716786 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:57.708529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35025 (local address 127.0.97.62:44471)
0927 22:09:57.708707 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:57.708713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:57.709055 (+ 342us) server_negotiation.cc:407] Connection header received
0927 22:09:57.709197 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:57.709201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:57.709275 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:57.709358 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:57.710112 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:57.710962 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:57.712161 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:57.712586 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:57.712694 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:57.716608 (+ 3914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35025: BlockingRecv error: recv got EOF from 127.0.97.1:35025 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 22:09:58 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:58 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:58.177033 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:58.168514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.2:55195)
0927 22:09:58.169029 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:58.169047 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:58.169171 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:58.169453 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:58.169457 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:58.169904 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:58.170124 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:58.170141 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:58.171321 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:58.171326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:58.172248 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:58.172259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:58.172924 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:58.172929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:58.173039 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:58.173712 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:58.173741 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:58.176799 (+ 3058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":280,"threads_started":1}
W20260927 22:09:58.177235 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:58.168587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55195 (local address 127.0.97.62:44471)
0927 22:09:58.168843 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:58.168848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:58.169126 (+ 278us) server_negotiation.cc:407] Connection header received
0927 22:09:58.169268 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:58.169272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:58.169333 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:58.169414 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:58.170269 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:58.171186 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:58.172380 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:58.172791 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:58.172882 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:58.176993 (+ 4111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55195: BlockingRecv error: recv got EOF from 127.0.97.2:55195 (error 108)
Metrics: {"server-negotiator.queue_time_us":151}
Sep 27 22:09:58 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:58 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:58.657897 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:58.641240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.3:51107)
0927 22:09:58.641796 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:58.641812 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:58.641930 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:58.642243 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:58.642246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:58.642679 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:58.642844 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:58.642852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:58.643808 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:58.643812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:58.644769 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:58.644780 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:58.645526 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:58.645532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:58.645701 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:58.654428 (+ 8727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:58.654450 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:58.657552 (+ 3102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":315,"threads_started":1}
W20260927 22:09:58.658004 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:58.641406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51107 (local address 127.0.97.62:44471)
0927 22:09:58.641579 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:58.641584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:58.641939 (+ 355us) server_negotiation.cc:407] Connection header received
0927 22:09:58.642043 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:58.642047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:58.642109 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:58.642197 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:58.642985 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:58.643689 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:58.644913 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:58.645357 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:58.645515 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:58.657768 (+ 12253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51107: BlockingRecv error: recv got EOF from 127.0.97.3:51107 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 22:09:58 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:58 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:58.727171 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:58.717875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.1:59995)
0927 22:09:58.718489 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:58.718505 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:58.718673 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:58.719129 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:58.719134 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:58.719556 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:58.719702 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:58.719711 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:58.720714 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:58.720718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:58.721961 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:58.721977 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:58.723022 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:58.723026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:58.723153 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:58.723786 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:58.723807 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:58.726811 (+ 3004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":308,"threads_started":1}
W20260927 22:09:58.727260 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:58.717975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59995 (local address 127.0.97.62:44471)
0927 22:09:58.718224 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:58.718230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:58.718613 (+ 383us) server_negotiation.cc:407] Connection header received
0927 22:09:58.718858 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:58.718865 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:58.718972 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:58.719082 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:58.719835 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:58.720591 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:58.722104 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:58.722890 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:58.722997 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:58.727023 (+ 4026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59995: BlockingRecv error: recv got EOF from 127.0.97.1:59995 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 27 22:09:59 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:59 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:59.186183 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:59.178037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.2:46373)
0927 22:09:59.178786 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:59.178808 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:59.178978 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:59.179349 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:59.179354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:59.179781 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:59.179970 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:59.179978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:59.180935 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:59.180939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:59.181823 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:59.181834 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:59.182471 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:59.182475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:59.182645 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:59.183260 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:59.183292 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:59.185957 (+ 2665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":445,"threads_started":1}
W20260927 22:09:59.186352 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:59.178128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46373 (local address 127.0.97.62:44471)
0927 22:09:59.178370 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:59.178377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:59.178902 (+ 525us) server_negotiation.cc:407] Connection header received
0927 22:09:59.179127 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:59.179131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:59.179193 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:59.179270 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:59.180116 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:59.180811 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:59.181944 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:59.182353 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:59.182442 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:59.186129 (+ 3687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46373: BlockingRecv error: recv got EOF from 127.0.97.2:46373 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 27 22:09:59 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:59 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:59.667887 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:59.659262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.3:41461)
0927 22:09:59.659823 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:59.659845 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:59.660041 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:59.660355 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:59.660358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:59.660709 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:59.660857 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:59.660865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:59.661986 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:59.661990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:59.663070 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:59.663109 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:59.664011 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:59.664017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:59.664226 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:59.664827 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:59.664855 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:59.667655 (+ 2800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":306,"threads_started":1}
W20260927 22:09:59.668110 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:59.659277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41461 (local address 127.0.97.62:44471)
0927 22:09:59.659518 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:59.659530 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:59.660046 (+ 516us) server_negotiation.cc:407] Connection header received
0927 22:09:59.660158 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:59.660162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:59.660227 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:59.660320 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:59.661012 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:59.661801 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:59.663261 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:59.663820 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:59.664003 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:59.667851 (+ 3848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41461: BlockingRecv error: recv got EOF from 127.0.97.3:41461 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 22:09:59 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:09:59 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:09:59.736598 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:09:59.728075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.1:38283)
0927 22:09:59.728705 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0927 22:09:59.728721 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:09:59.728834 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:09:59.729173 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:09:59.729177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:09:59.729560 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:09:59.729715 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:59.729724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:59.730787 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:59.730791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:59.731774 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:09:59.731785 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:09:59.732486 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:09:59.732491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:09:59.732600 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:59.733254 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:09:59.733273 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:09:59.736265 (+ 2992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":405,"threads_started":1}
W20260927 22:09:59.736689 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:09:59.728151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38283 (local address 127.0.97.62:44471)
0927 22:09:59.728415 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 22:09:59.728425 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:09:59.728830 (+ 405us) server_negotiation.cc:407] Connection header received
0927 22:09:59.728963 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:09:59.728967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:09:59.729043 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:09:59.729135 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:09:59.729840 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:59.730654 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:59.731906 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:09:59.732325 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:09:59.732421 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:09:59.736458 (+ 4037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38283: BlockingRecv error: recv got EOF from 127.0.97.1:38283 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 22:10:00 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:10:00 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:10:00.195770 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:00.187321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.2:49457)
0927 22:10:00.187861 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0927 22:10:00.187878 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:10:00.188039 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:10:00.188373 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:10:00.188377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:10:00.188773 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:10:00.188921 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:00.188929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:00.189948 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:00.189954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:00.190932 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:00.190943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:00.191662 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:00.191669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:00.191831 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:00.192436 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:10:00.192453 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:10:00.195514 (+ 3061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":244,"threads_started":1}
W20260927 22:10:00.196036 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:10:00.187550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49457 (local address 127.0.97.62:44471)
0927 22:10:00.187752 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:00.187759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:00.187971 (+ 212us) server_negotiation.cc:407] Connection header received
0927 22:10:00.188166 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:00.188172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:00.188238 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:00.188348 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:00.189064 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:00.189812 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:00.191067 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:00.191477 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:00.191608 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:00.195745 (+ 4137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49457: BlockingRecv error: recv got EOF from 127.0.97.2:49457 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 22:10:00 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:10:00 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:10:00.677579 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:00.669097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.3:39295)
0927 22:10:00.669823 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0927 22:10:00.669839 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:10:00.669997 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:10:00.670344 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:10:00.670349 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:10:00.670818 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:10:00.670973 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:00.670983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:00.671977 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:00.671983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:00.672940 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:00.672951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:00.673699 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:00.673706 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:00.673884 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:00.674455 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:10:00.674472 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:10:00.677228 (+ 2756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":457,"threads_started":1}
W20260927 22:10:00.677659 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:10:00.669152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39295 (local address 127.0.97.62:44471)
0927 22:10:00.669410 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:00.669419 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:00.669937 (+ 518us) server_negotiation.cc:407] Connection header received
0927 22:10:00.670139 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:00.670144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:00.670213 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:00.670318 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:00.671122 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:00.671835 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:00.673094 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:00.673520 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:00.673663 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:00.677426 (+ 3763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39295: BlockingRecv error: recv got EOF from 127.0.97.3:39295 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 27 22:10:00 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:10:00 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:10:00.747268 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:00.737939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.1:35765)
0927 22:10:00.738496 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 22:10:00.738521 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:10:00.738746 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:10:00.739188 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:10:00.739193 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:10:00.739610 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:10:00.739799 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:00.739809 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:00.740940 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:00.740947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:00.741911 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:00.741924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:00.742858 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:00.742864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:00.743075 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:00.743815 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:10:00.743835 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:10:00.746870 (+ 3035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":296,"threads_started":1}
W20260927 22:10:00.747394 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:10:00.738094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35765 (local address 127.0.97.62:44471)
0927 22:10:00.738488 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:00.738494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:00.738640 (+ 146us) server_negotiation.cc:407] Connection header received
0927 22:10:00.738905 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:00.738914 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:00.739027 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:00.739155 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:00.739970 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:00.740775 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:00.742082 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:00.742701 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:00.742849 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:00.747102 (+ 4253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35765: BlockingRecv error: recv got EOF from 127.0.97.1:35765 (error 108)
Metrics: {"server-negotiator.queue_time_us":175}
Sep 27 22:10:01 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:10:01 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:10:01.205529 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:01.196946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.2:58243)
0927 22:10:01.197491 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0927 22:10:01.197508 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:10:01.197630 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:10:01.197941 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:10:01.197944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:10:01.198286 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:10:01.198507 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:01.198516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:01.199615 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:01.199619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:01.200526 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:01.200537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:01.201295 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:01.201299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:01.201445 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:01.202147 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:10:01.202164 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:10:01.205216 (+ 3052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":290,"threads_started":1}
W20260927 22:10:01.205811 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:10:01.197050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58243 (local address 127.0.97.62:44471)
0927 22:10:01.197306 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:01.197313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:01.197596 (+ 283us) server_negotiation.cc:407] Connection header received
0927 22:10:01.197732 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:01.197736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:01.197807 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:01.197902 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:01.198711 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:01.199489 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:01.200675 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:01.201155 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:01.201281 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:01.205532 (+ 4251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58243: BlockingRecv error: recv got EOF from 127.0.97.2:58243 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 22:10:01 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:10:01 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:10:01.687147 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:01.678749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.3:41631)
0927 22:10:01.679258 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0927 22:10:01.679274 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:10:01.679430 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:10:01.679780 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:10:01.679784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:10:01.680167 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:10:01.680360 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:01.680375 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:01.681676 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:01.681681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:01.682848 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:01.682859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:01.683487 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:01.683490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:01.683600 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:01.684200 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:10:01.684216 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:10:01.686900 (+ 2684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":294,"threads_started":1}
W20260927 22:10:01.687395 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:10:01.678845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41631 (local address 127.0.97.62:44471)
0927 22:10:01.679114 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:01.679130 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:01.679357 (+ 227us) server_negotiation.cc:407] Connection header received
0927 22:10:01.679558 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:01.679564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:01.679640 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:01.679748 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:01.680539 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:01.681499 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:01.682964 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:01.683372 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:01.683475 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:01.687103 (+ 3628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41631: BlockingRecv error: recv got EOF from 127.0.97.3:41631 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 22:10:01 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:10:01 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:10:01.757361 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:01.748565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.1:38055)
0927 22:10:01.749096 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 22:10:01.749114 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:10:01.749295 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:10:01.749705 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:10:01.749711 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:10:01.750117 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:10:01.750298 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:01.750306 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:01.751375 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:01.751381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:01.752282 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:01.752293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:01.753116 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:01.753122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:01.753317 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:01.754229 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:10:01.754254 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:10:01.757107 (+ 2853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":306,"threads_started":1}
W20260927 22:10:01.757560 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:10:01.748636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38055 (local address 127.0.97.62:44471)
0927 22:10:01.748904 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:01.748913 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:01.749211 (+ 298us) server_negotiation.cc:407] Connection header received
0927 22:10:01.749430 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:01.749438 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:01.749531 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:01.749639 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:01.750450 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:01.751243 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:01.752423 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:01.752917 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:01.753053 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:01.757317 (+ 4264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38055: BlockingRecv error: recv got EOF from 127.0.97.1:38055 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 27 22:10:02 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:10:02 dist-test-slave-mp03 krb5kdc[8616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:10:02.215462 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:02.207012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.2:57231)
0927 22:10:02.207575 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 22:10:02.207596 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:10:02.207759 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:10:02.208125 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:10:02.208128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:10:02.208574 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:10:02.208772 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:02.208781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:02.209848 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:02.209854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:02.210822 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:02.210833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:02.211541 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:02.211547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:02.211712 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:02.212304 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:10:02.212321 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:10:02.215196 (+ 2875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:44471: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":272,"threads_started":1}
W20260927 22:10:02.215653 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:10:02.207064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57231 (local address 127.0.97.62:44471)
0927 22:10:02.207301 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:02.207312 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:02.207749 (+ 437us) server_negotiation.cc:407] Connection header received
0927 22:10:02.207920 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:02.207924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:02.207993 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:02.208085 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:02.208916 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:02.209678 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:02.210958 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:02.211386 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:02.211525 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:02.215392 (+ 3867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57231: BlockingRecv error: recv got EOF from 127.0.97.2:57231 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:10:02.505434 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 9204
I20260927 22:10:02.517756 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:02.506508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53782 (local address 127.0.97.1:41701)
0927 22:10:02.507251 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:02.507259 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:02.507286 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:10:02.507355 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:02.507361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:02.507670 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:02.507830 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:02.509358 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:02.510367 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:02.511606 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:02.512168 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:02.512263 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:02.514718 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:02.514726 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:02.514741 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:10:02.514800 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:02.517282 (+ 2482us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 22:10:02.517529 (+ 247us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53782: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":469,"threads_started":1}
W20260927 22:10:02.517892 9664 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53782: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 22:10:02.518013 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 9204: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:41701: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 22:10:02.518222 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 9204
I20260927 22:10:02.534981 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 9355
I20260927 22:10:02.546994 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:02.536121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54746 (local address 127.0.97.2:46503)
0927 22:10:02.536684 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:02.536692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:02.536721 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:10:02.536780 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:02.536786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:02.537051 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:02.537204 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:02.538426 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:02.539528 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:02.540606 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:02.541118 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:02.541211 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:02.543804 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:02.543849 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:02.543863 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:10:02.543902 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:02.546413 (+ 2511us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 22:10:02.546698 (+ 285us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54746: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":270,"threads_started":1}
W20260927 22:10:02.547104 9665 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54746: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 22:10:02.547097 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 9355: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:46503: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 22:10:02.547254 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 9355
I20260927 22:10:02.563773 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 9486
I20260927 22:10:02.575970 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:02.564884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34288 (local address 127.0.97.3:36523)
0927 22:10:02.565445 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:02.565452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:02.565479 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:10:02.565541 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:02.565547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:02.565768 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:02.565934 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:02.567379 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:02.568433 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:02.569537 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:02.570070 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:02.570162 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:02.572798 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:02.572806 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:02.572820 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:10:02.572855 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:02.575438 (+ 2583us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 22:10:02.575678 (+ 240us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34288: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":311,"threads_started":1}
W20260927 22:10:02.576077 9666 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34288: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 22:10:02.576117 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 9486: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:36523: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 22:10:02.576280 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 9486
I20260927 22:10:02.592208 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 9135
I20260927 22:10:02.593518 9197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:10:02.707592 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 9135
2026-09-27T22:10:02Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21198 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T22:10:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:10:02Z Disabled control of system clock
I20260927 22:10:02.785326 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39609
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39609
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 22:10:03.119585 9671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:10:03.119884 9671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:10:03.119961 9671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:10:03.127264 9671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:10:03.127377 9671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:10:03.127416 9671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:10:03.127466 9671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:10:03.127511 9671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:10:03.136686 9671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39609
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:10:03.137964 9671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:10:03.139724 9671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:10:03.151659 9677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:03.151641 9676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:03.151645 9679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:10:03.152134 9671 server_base.cc:1034] running on GCE node
I20260927 22:10:03.153033 9671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:10:03.154466 9671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:10:03.155725 9671 hybrid_clock.cc:648] HybridClock initialized: now 1790547003155698 us; error 59 us; skew 500 ppm
I20260927 22:10:03.156102 9671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:10:03.158712 9671 webserver.cc:460] Webserver started at http://127.0.97.62:35705/ using document root <none> and password file <none>
I20260927 22:10:03.159492 9671 fs_manager.cc:359] Metadata directory not provided
I20260927 22:10:03.159634 9671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:10:03.159969 9671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:10:03.162986 9671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "caadca7075704ec78a6054f67314223e"
format_stamp: "Formatted at 2026-09-27 22:10:03 on dist-test-slave-mp03"
server_key: "b52ca2ad9015ad426603ef077540762c"
server_key_iv: "05a64f2827df86f1d64054e8490d2bef"
server_key_version: "encryptionkey@0"
I20260927 22:10:03.163807 9671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "caadca7075704ec78a6054f67314223e"
format_stamp: "Formatted at 2026-09-27 22:10:03 on dist-test-slave-mp03"
server_key: "b52ca2ad9015ad426603ef077540762c"
server_key_iv: "05a64f2827df86f1d64054e8490d2bef"
server_key_version: "encryptionkey@0"
I20260927 22:10:03.169307 9671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:10:03.173169 9685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:03.174774 9671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:10:03.174971 9671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "caadca7075704ec78a6054f67314223e"
format_stamp: "Formatted at 2026-09-27 22:10:03 on dist-test-slave-mp03"
server_key: "b52ca2ad9015ad426603ef077540762c"
server_key_iv: "05a64f2827df86f1d64054e8490d2bef"
server_key_version: "encryptionkey@0"
I20260927 22:10:03.175220 9671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:10:03.240602 9671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:10:03.241654 9671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:10:03.241997 9671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:10:03.263391 9671 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39609
I20260927 22:10:03.263413 9736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39609 every 8 connection(s)
I20260927 22:10:03.264869 9671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:10:03.267366 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 9671
I20260927 22:10:03.267578 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:10:03.268308 388 external_mini_cluster.cc:1428] Setting key 9f068887ba3f87684c29c52d5f6a5c06
I20260927 22:10:03.268859 9737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:10:03.282680 9737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caadca7075704ec78a6054f67314223e: Bootstrap starting.
I20260927 22:10:03.283967 9740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:03.270643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57238 (local address 127.0.97.62:39609)
0927 22:10:03.271682 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:03.271697 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:03.271754 (+ 57us) server_negotiation.cc:407] Connection header received
0927 22:10:03.273823 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:03.273852 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:03.274466 (+ 614us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:03.274948 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:03.275481 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:03.277462 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:03.279008 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:03.279944 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:03.280172 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:03.281498 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:03.281505 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:03.281519 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:10:03.282069 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:03.282162 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:10:03.282187 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:10:03.282321 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:03.282568 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 22:10:03.282971 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":310,"threads_started":1}
I20260927 22:10:03.285780 9737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P caadca7075704ec78a6054f67314223e: Neither blocks nor log segments found. Creating new log.
I20260927 22:10:03.286906 9737 log.cc:824] T 00000000000000000000000000000000 P caadca7075704ec78a6054f67314223e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:10:03.289522 9737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caadca7075704ec78a6054f67314223e: No bootstrap required, opened a new log
I20260927 22:10:03.296542 9737 raft_consensus.cc:348] T 00000000000000000000000000000000 P caadca7075704ec78a6054f67314223e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caadca7075704ec78a6054f67314223e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39609 } }
I20260927 22:10:03.296890 9737 raft_consensus.cc:374] T 00000000000000000000000000000000 P caadca7075704ec78a6054f67314223e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:10:03.297008 9737 raft_consensus.cc:729] T 00000000000000000000000000000000 P caadca7075704ec78a6054f67314223e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caadca7075704ec78a6054f67314223e, State: Initialized, Role: FOLLOWER
I20260927 22:10:03.297700 9737 consensus_queue.cc:260] T 00000000000000000000000000000000 P caadca7075704ec78a6054f67314223e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caadca7075704ec78a6054f67314223e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39609 } }
I20260927 22:10:03.298030 9737 raft_consensus.cc:388] T 00000000000000000000000000000000 P caadca7075704ec78a6054f67314223e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:10:03.298207 9737 raft_consensus.cc:482] T 00000000000000000000000000000000 P caadca7075704ec78a6054f67314223e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:10:03.298403 9737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P caadca7075704ec78a6054f67314223e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:10:03.300709 9737 raft_consensus.cc:504] T 00000000000000000000000000000000 P caadca7075704ec78a6054f67314223e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caadca7075704ec78a6054f67314223e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39609 } }
I20260927 22:10:03.301213 9737 leader_election.cc:302] T 00000000000000000000000000000000 P caadca7075704ec78a6054f67314223e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: caadca7075704ec78a6054f67314223e; no voters:
I20260927 22:10:03.301837 9737 leader_election.cc:288] T 00000000000000000000000000000000 P caadca7075704ec78a6054f67314223e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:10:03.302100 9742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P caadca7075704ec78a6054f67314223e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:10:03.302996 9742 raft_consensus.cc:686] T 00000000000000000000000000000000 P caadca7075704ec78a6054f67314223e [term 1 LEADER]: Becoming Leader. State: Replica: caadca7075704ec78a6054f67314223e, State: Running, Role: LEADER
I20260927 22:10:03.303548 9742 consensus_queue.cc:237] T 00000000000000000000000000000000 P caadca7075704ec78a6054f67314223e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caadca7075704ec78a6054f67314223e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39609 } }
I20260927 22:10:03.304006 9737 sys_catalog.cc:564] T 00000000000000000000000000000000 P caadca7075704ec78a6054f67314223e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:10:03.306349 9744 sys_catalog.cc:455] T 00000000000000000000000000000000 P caadca7075704ec78a6054f67314223e [sys.catalog]: SysCatalogTable state changed. Reason: New leader caadca7075704ec78a6054f67314223e. Latest consensus state: current_term: 1 leader_uuid: "caadca7075704ec78a6054f67314223e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caadca7075704ec78a6054f67314223e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39609 } } }
I20260927 22:10:03.306802 9744 sys_catalog.cc:458] T 00000000000000000000000000000000 P caadca7075704ec78a6054f67314223e [sys.catalog]: This master's current role is: LEADER
I20260927 22:10:03.306871 9743 sys_catalog.cc:455] T 00000000000000000000000000000000 P caadca7075704ec78a6054f67314223e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "caadca7075704ec78a6054f67314223e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caadca7075704ec78a6054f67314223e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39609 } } }
I20260927 22:10:03.307598 9743 sys_catalog.cc:458] T 00000000000000000000000000000000 P caadca7075704ec78a6054f67314223e [sys.catalog]: This master's current role is: LEADER
I20260927 22:10:03.310102 9747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:10:03.315448 9747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:10:03.324383 9747 catalog_manager.cc:1347] Generated new cluster ID: 669bc051f0a04e059f0c15d2c2070c82
I20260927 22:10:03.324529 9747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:10:03.342034 9747 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:10:03.343524 9747 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:10:03.353056 9747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P caadca7075704ec78a6054f67314223e: Generated new TSK 0
I20260927 22:10:03.353922 9747 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:10:03.360750 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39609
--builtin_ntp_servers=127.0.97.20:37583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 22:10:03.693259 9761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:10:03.693635 9761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:10:03.693778 9761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:10:03.701259 9761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:10:03.701408 9761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:10:03.701570 9761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:10:03.711990 9761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:10:03.713330 9761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:10:03.715236 9761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:10:03.728675 9769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:03.728749 9766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:03.728708 9767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:10:03.729533 9761 server_base.cc:1034] running on GCE node
I20260927 22:10:03.730144 9761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:10:03.731225 9761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:10:03.732491 9761 hybrid_clock.cc:648] HybridClock initialized: now 1790547003732443 us; error 60 us; skew 500 ppm
I20260927 22:10:03.732825 9761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:10:03.739732 9761 webserver.cc:460] Webserver started at http://127.0.97.1:44107/ using document root <none> and password file <none>
I20260927 22:10:03.740479 9761 fs_manager.cc:359] Metadata directory not provided
I20260927 22:10:03.740615 9761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:10:03.740947 9761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:10:03.743861 9761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "02ff8b2537444a0ebaa9296c463e128c"
format_stamp: "Formatted at 2026-09-27 22:10:03 on dist-test-slave-mp03"
server_key: "d5c6ad60407494c02c2767844cc8574a"
server_key_iv: "b3affa1767912f9191cababc53b64b6e"
server_key_version: "encryptionkey@0"
I20260927 22:10:03.744675 9761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "02ff8b2537444a0ebaa9296c463e128c"
format_stamp: "Formatted at 2026-09-27 22:10:03 on dist-test-slave-mp03"
server_key: "d5c6ad60407494c02c2767844cc8574a"
server_key_iv: "b3affa1767912f9191cababc53b64b6e"
server_key_version: "encryptionkey@0"
I20260927 22:10:03.749804 9761 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:10:03.753358 9775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:03.754932 9761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:10:03.755106 9761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "02ff8b2537444a0ebaa9296c463e128c"
format_stamp: "Formatted at 2026-09-27 22:10:03 on dist-test-slave-mp03"
server_key: "d5c6ad60407494c02c2767844cc8574a"
server_key_iv: "b3affa1767912f9191cababc53b64b6e"
server_key_version: "encryptionkey@0"
I20260927 22:10:03.755327 9761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:10:03.773782 9761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:10:03.774730 9761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:10:03.775008 9761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:10:03.776008 9761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:10:03.777491 9761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:10:03.777621 9761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:03.777761 9761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:10:03.777851 9761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:03.811198 9761 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42023
I20260927 22:10:03.811211 9887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42023 every 8 connection(s)
I20260927 22:10:03.812527 9761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:10:03.815748 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 9761
I20260927 22:10:03.815928 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:10:03.816799 388 external_mini_cluster.cc:1428] Setting key ffec874a6a5ebeea060d4dae66e27d60
I20260927 22:10:03.819542 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39609
--builtin_ntp_servers=127.0.97.20:37583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 22:10:03.826558 9890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:03.815327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53397 (local address 127.0.97.62:39609)
0927 22:10:03.815813 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:03.815818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:03.817252 (+ 1434us) server_negotiation.cc:407] Connection header received
0927 22:10:03.819420 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:03.819426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:03.819527 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:03.819650 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:03.820809 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:03.821543 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:03.823257 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:03.823712 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:03.823860 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:03.825535 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:03.825540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:03.825543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:10:03.825561 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:03.825621 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:10:03.825633 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:10:03.825707 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:03.826006 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 22:10:03.826101 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":262,"threads_started":1}
I20260927 22:10:03.827629 9891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:03.815938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39609 (local address 127.0.97.1:53397)
0927 22:10:03.817069 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0927 22:10:03.817107 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 22:10:03.819122 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:10:03.819900 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:10:03.819909 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:10:03.820009 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:10:03.820621 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:03.820638 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:03.821700 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:03.821708 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:03.823100 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:03.823117 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:03.823968 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:03.823976 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:03.824222 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:03.825052 (+ 830us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:10:03.825058 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:10:03.825116 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:10:03.825228 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:10:03.825741 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:10:03.825755 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:10:03.825766 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 22:10:03.825996 (+ 230us) client_negotiation.cc:240] Negotiation successful
0927 22:10:03.826387 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":378,"threads_started":1}
I20260927 22:10:03.829403 9888 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39609
I20260927 22:10:03.829730 9888 heartbeater.cc:464] Registering TS with master...
I20260927 22:10:03.830570 9888 heartbeater.cc:511] Master 127.0.97.62:39609 requested a full tablet report, sending...
I20260927 22:10:03.832940 9702 ts_manager.cc:194] Registered new tserver with Master: 02ff8b2537444a0ebaa9296c463e128c (127.0.97.1:42023)
I20260927 22:10:03.835295 9702 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:53397
W20260927 22:10:04.150216 9892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:10:04.150506 9892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:10:04.150637 9892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:10:04.157799 9892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:10:04.157900 9892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:10:04.158002 9892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:10:04.167174 9892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:10:04.168444 9892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:10:04.170120 9892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:10:04.183581 9900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:04.183609 9898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:04.183656 9897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:10:04.184406 9892 server_base.cc:1034] running on GCE node
I20260927 22:10:04.184995 9892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:10:04.185928 9892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:10:04.187155 9892 hybrid_clock.cc:648] HybridClock initialized: now 1790547004187080 us; error 57 us; skew 500 ppm
I20260927 22:10:04.187507 9892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:10:04.189822 9892 webserver.cc:460] Webserver started at http://127.0.97.2:36217/ using document root <none> and password file <none>
I20260927 22:10:04.190568 9892 fs_manager.cc:359] Metadata directory not provided
I20260927 22:10:04.190743 9892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:10:04.191088 9892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:10:04.193998 9892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "f3b45c8568e640299ad12980a2b80981"
format_stamp: "Formatted at 2026-09-27 22:10:04 on dist-test-slave-mp03"
server_key: "24ba72facbd6a500d4568042deb31b25"
server_key_iv: "cb17db1997b9a55c09f6f03895f2cd44"
server_key_version: "encryptionkey@0"
I20260927 22:10:04.194850 9892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f3b45c8568e640299ad12980a2b80981"
format_stamp: "Formatted at 2026-09-27 22:10:04 on dist-test-slave-mp03"
server_key: "24ba72facbd6a500d4568042deb31b25"
server_key_iv: "cb17db1997b9a55c09f6f03895f2cd44"
server_key_version: "encryptionkey@0"
I20260927 22:10:04.200006 9892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 22:10:04.203464 9906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:04.204980 9892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:10:04.205155 9892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "f3b45c8568e640299ad12980a2b80981"
format_stamp: "Formatted at 2026-09-27 22:10:04 on dist-test-slave-mp03"
server_key: "24ba72facbd6a500d4568042deb31b25"
server_key_iv: "cb17db1997b9a55c09f6f03895f2cd44"
server_key_version: "encryptionkey@0"
I20260927 22:10:04.205402 9892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:10:04.217502 9892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:10:04.218313 9892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:10:04.218581 9892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:10:04.219606 9892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:10:04.220976 9892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:10:04.221098 9892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:04.221221 9892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:10:04.221310 9892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:04.253974 9892 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42855
I20260927 22:10:04.254045 10018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42855 every 8 connection(s)
I20260927 22:10:04.255373 9892 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:10:04.263648 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 9892
I20260927 22:10:04.263818 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:10:04.264328 388 external_mini_cluster.cc:1428] Setting key 0e9058d0e1fc8f2afe7caa68f499310f
I20260927 22:10:04.267235 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39609
--builtin_ntp_servers=127.0.97.20:37583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 22:10:04.270033 9890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:04.257893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38069 (local address 127.0.97.62:39609)
0927 22:10:04.258066 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:04.258071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:04.259750 (+ 1679us) server_negotiation.cc:407] Connection header received
0927 22:10:04.261604 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:04.261609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:04.261692 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:04.261807 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:04.263038 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:04.264253 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:04.266524 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:04.267242 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:04.267395 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:04.269063 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:04.269072 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:04.269079 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:10:04.269104 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:04.269147 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:10:04.269179 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:10:04.269272 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:04.269514 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 22:10:04.269626 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:10:04.271209 10021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:04.258524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39609 (local address 127.0.97.2:38069)
0927 22:10:04.259596 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0927 22:10:04.259631 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:10:04.261335 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:10:04.262030 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:10:04.262038 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:10:04.262128 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:10:04.262805 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:04.262853 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:04.264429 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:04.264434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:04.266375 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:04.266389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:04.267496 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:04.267506 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:04.267817 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:04.268683 (+ 866us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:10:04.268687 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:10:04.268743 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:10:04.268847 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:10:04.269289 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:10:04.269298 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:10:04.269307 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 22:10:04.269489 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 22:10:04.269900 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":260,"threads_started":1}
I20260927 22:10:04.272754 10019 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39609
I20260927 22:10:04.273078 10019 heartbeater.cc:464] Registering TS with master...
I20260927 22:10:04.274016 10019 heartbeater.cc:511] Master 127.0.97.62:39609 requested a full tablet report, sending...
I20260927 22:10:04.275619 9702 ts_manager.cc:194] Registered new tserver with Master: f3b45c8568e640299ad12980a2b80981 (127.0.97.2:42855)
I20260927 22:10:04.276913 9702 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:38069
W20260927 22:10:04.608551 10022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:10:04.608956 10022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:10:04.609118 10022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:10:04.616714 10022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:10:04.616827 10022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:10:04.617022 10022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:10:04.626137 10022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:10:04.627476 10022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:10:04.629268 10022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:10:04.642536 10027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:04.642500 10030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:04.642629 10028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:10:04.643177 10022 server_base.cc:1034] running on GCE node
I20260927 22:10:04.643837 10022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:10:04.644804 10022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:10:04.646023 10022 hybrid_clock.cc:648] HybridClock initialized: now 1790547004645965 us; error 56 us; skew 500 ppm
I20260927 22:10:04.646344 10022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:10:04.648723 10022 webserver.cc:460] Webserver started at http://127.0.97.3:37907/ using document root <none> and password file <none>
I20260927 22:10:04.649439 10022 fs_manager.cc:359] Metadata directory not provided
I20260927 22:10:04.649570 10022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:10:04.649893 10022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:10:04.652808 10022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "6687456a98454ac8a3d97aa3a2db8dcf"
format_stamp: "Formatted at 2026-09-27 22:10:04 on dist-test-slave-mp03"
server_key: "12806f4f25242d2292ab93f5cdaa4cfa"
server_key_iv: "b46aa7a7f52b75b86fee9050d36d869b"
server_key_version: "encryptionkey@0"
I20260927 22:10:04.653681 10022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6687456a98454ac8a3d97aa3a2db8dcf"
format_stamp: "Formatted at 2026-09-27 22:10:04 on dist-test-slave-mp03"
server_key: "12806f4f25242d2292ab93f5cdaa4cfa"
server_key_iv: "b46aa7a7f52b75b86fee9050d36d869b"
server_key_version: "encryptionkey@0"
I20260927 22:10:04.659304 10022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:10:04.663316 10036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:04.664746 10022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:10:04.664937 10022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "6687456a98454ac8a3d97aa3a2db8dcf"
format_stamp: "Formatted at 2026-09-27 22:10:04 on dist-test-slave-mp03"
server_key: "12806f4f25242d2292ab93f5cdaa4cfa"
server_key_iv: "b46aa7a7f52b75b86fee9050d36d869b"
server_key_version: "encryptionkey@0"
I20260927 22:10:04.665174 10022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:10:04.681185 10022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:10:04.682078 10022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:10:04.682361 10022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:10:04.683421 10022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:10:04.684930 10022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:10:04.685050 10022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:04.685189 10022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:10:04.685274 10022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:04.719302 10022 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35271
I20260927 22:10:04.719333 10148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35271 every 8 connection(s)
I20260927 22:10:04.720625 10022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:10:04.722143 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 10022
I20260927 22:10:04.722294 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:10:04.722784 388 external_mini_cluster.cc:1428] Setting key 38aa45650f0e0708b881b9dfe78066d0
I20260927 22:10:04.734514 9890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:04.724155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40025 (local address 127.0.97.62:39609)
0927 22:10:04.724358 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:04.724364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:04.726089 (+ 1725us) server_negotiation.cc:407] Connection header received
0927 22:10:04.728289 (+ 2200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:04.728294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:04.728383 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:04.728484 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:04.729508 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:04.730225 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:04.731826 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:04.732262 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:04.732390 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:04.733748 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:04.733753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:04.733756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:10:04.733773 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:04.733801 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:10:04.733812 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:10:04.733905 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:04.734117 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 22:10:04.734206 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:10:04.735582 10151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:04.724870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39609 (local address 127.0.97.3:40025)
0927 22:10:04.725929 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0927 22:10:04.725981 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 22:10:04.728045 (+ 2064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:10:04.728716 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:10:04.728724 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:10:04.728808 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:10:04.729359 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:04.729375 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:04.730367 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:04.730373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:04.731688 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:04.731700 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:04.732487 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:04.732494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:04.732721 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:04.733430 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:10:04.733432 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:10:04.733478 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:10:04.733556 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:10:04.733904 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:10:04.733917 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:10:04.733927 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 22:10:04.734111 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 22:10:04.734494 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":309,"threads_started":1}
I20260927 22:10:04.737196 10149 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39609
I20260927 22:10:04.737488 10149 heartbeater.cc:464] Registering TS with master...
I20260927 22:10:04.738296 10149 heartbeater.cc:511] Master 127.0.97.62:39609 requested a full tablet report, sending...
I20260927 22:10:04.739755 9702 ts_manager.cc:194] Registered new tserver with Master: 6687456a98454ac8a3d97aa3a2db8dcf (127.0.97.3:35271)
I20260927 22:10:04.740535 9702 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:40025
I20260927 22:10:04.750483 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 22:10:04.756932 9890 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:10:04.755203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57248 (local address 127.0.97.62:39609)
0927 22:10:04.755403 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:04.755407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:04.755966 (+ 559us) server_negotiation.cc:407] Connection header received
0927 22:10:04.756148 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:04.756152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:04.756222 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:04.756310 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:04.756707 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57248: BlockingRecv error: recv got EOF from 127.0.0.1:57248 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:10:04.757501 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 9761
I20260927 22:10:04.765796 10158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:04.758567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55626 (local address 127.0.97.1:42023)
0927 22:10:04.759346 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:04.759353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:04.759381 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:10:04.759441 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:04.759449 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:04.759700 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:04.759828 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:04.760308 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:04.761405 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:04.762683 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:04.763283 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:04.763396 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:04.764416 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:04.764425 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:04.764438 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:10:04.765087 (+ 649us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:04.765136 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:10:04.765155 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:10:04.765251 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:04.765363 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 22:10:04.765486 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":455,"threads_started":1}
I20260927 22:10:04.767172 9883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:10:04.847813 9888 heartbeater.cc:503] Master 127.0.97.62:39609 was elected leader, sending a full tablet report...
I20260927 22:10:04.849848 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 9761
I20260927 22:10:04.866968 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 9892
I20260927 22:10:04.874914 10160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:04.867889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34348 (local address 127.0.97.2:42855)
0927 22:10:04.868401 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:04.868408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:04.868433 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:10:04.868493 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:04.868499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:04.868789 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:04.868972 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:04.869372 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:04.870438 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:04.871697 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:04.872261 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:04.872400 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:04.873442 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:04.873451 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:04.873464 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:10:04.874070 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:04.874116 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:10:04.874134 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:10:04.874267 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:04.874417 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 22:10:04.874564 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":286,"threads_started":1}
I20260927 22:10:04.876264 10014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:10:04.958737 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 9892
I20260927 22:10:04.975559 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 10022
I20260927 22:10:04.983909 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:04.976641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41942 (local address 127.0.97.3:35271)
0927 22:10:04.977308 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:04.977315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:04.977345 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:10:04.977408 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:04.977414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:04.977675 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:04.977814 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:04.978233 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:04.979348 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:04.980742 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:04.981311 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:04.981409 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:04.982363 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:04.982372 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:04.982387 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:10:04.983179 (+ 792us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:04.983233 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:10:04.983252 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:10:04.983343 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:04.983470 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 22:10:04.983601 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":380,"threads_started":1}
I20260927 22:10:04.985131 10144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:10:05.067945 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 10022
I20260927 22:10:05.085886 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 9671
I20260927 22:10:05.087185 9732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:10:05.187520 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 9671
2026-09-27T22:10:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2473 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T22:10:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:10:05Z Disabled control of system clock
I20260927 22:10:05.267330 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41917
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 22:10:05.605185 10168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:10:05.605494 10168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:10:05.605579 10168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:10:05.613121 10168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:10:05.613234 10168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:10:05.613271 10168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:10:05.613320 10168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:10:05.613363 10168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:10:05.622982 10168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41917
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:10:05.624763 10168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:10:05.626520 10168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:10:05.640961 10174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:05.640964 10173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:05.640992 10176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:10:05.642023 10168 server_base.cc:1034] running on GCE node
I20260927 22:10:05.643030 10168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:10:05.644502 10168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:10:05.645726 10168 hybrid_clock.cc:648] HybridClock initialized: now 1790547005645677 us; error 41 us; skew 500 ppm
I20260927 22:10:05.646103 10168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:10:05.648546 10168 webserver.cc:460] Webserver started at http://127.0.97.62:37795/ using document root <none> and password file <none>
I20260927 22:10:05.649235 10168 fs_manager.cc:359] Metadata directory not provided
I20260927 22:10:05.649328 10168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:10:05.649598 10168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:10:05.652654 10168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "6a2b62863059419babc733f544e21401"
format_stamp: "Formatted at 2026-09-27 22:10:05 on dist-test-slave-mp03"
server_key: "e2adf85829e1720015b2f6c425bc899c"
server_key_iv: "33463b500916d0a6f195541e79251214"
server_key_version: "encryptionkey@0"
I20260927 22:10:05.653394 10168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "6a2b62863059419babc733f544e21401"
format_stamp: "Formatted at 2026-09-27 22:10:05 on dist-test-slave-mp03"
server_key: "e2adf85829e1720015b2f6c425bc899c"
server_key_iv: "33463b500916d0a6f195541e79251214"
server_key_version: "encryptionkey@0"
I20260927 22:10:05.658828 10168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260927 22:10:05.662529 10182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:05.664031 10168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:10:05.664175 10168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "6a2b62863059419babc733f544e21401"
format_stamp: "Formatted at 2026-09-27 22:10:05 on dist-test-slave-mp03"
server_key: "e2adf85829e1720015b2f6c425bc899c"
server_key_iv: "33463b500916d0a6f195541e79251214"
server_key_version: "encryptionkey@0"
I20260927 22:10:05.664361 10168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:10:05.670312 10168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:10:05.671128 10168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:10:05.671377 10168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:10:05.690097 10168 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41917
I20260927 22:10:05.690140 10233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41917 every 8 connection(s)
I20260927 22:10:05.691535 10168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:10:05.693639 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 10168
I20260927 22:10:05.693859 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:10:05.694677 388 external_mini_cluster.cc:1428] Setting key c887d27203cb582a3f98dcee0f96a3b6
I20260927 22:10:05.696210 10234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:10:05.703455 10237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:05.696842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50062 (local address 127.0.97.62:41917)
0927 22:10:05.697923 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:05.697935 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:05.697978 (+ 43us) server_negotiation.cc:407] Connection header received
0927 22:10:05.699605 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:05.699629 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:05.700092 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:05.700525 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:05.701657 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:05.701663 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:05.701676 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:10:05.701690 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:05.701765 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:10:05.701784 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:10:05.702027 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:05.702170 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 22:10:05.702445 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":471,"threads_started":1}
I20260927 22:10:05.709553 10234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a2b62863059419babc733f544e21401: Bootstrap starting.
I20260927 22:10:05.712550 10234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a2b62863059419babc733f544e21401: Neither blocks nor log segments found. Creating new log.
I20260927 22:10:05.713614 10234 log.cc:824] T 00000000000000000000000000000000 P 6a2b62863059419babc733f544e21401: Log is configured to *not* fsync() on all Append() calls
I20260927 22:10:05.716804 10234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a2b62863059419babc733f544e21401: No bootstrap required, opened a new log
I20260927 22:10:05.724097 10234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a2b62863059419babc733f544e21401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a2b62863059419babc733f544e21401" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41917 } }
I20260927 22:10:05.724478 10234 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a2b62863059419babc733f544e21401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:10:05.724628 10234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a2b62863059419babc733f544e21401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2b62863059419babc733f544e21401, State: Initialized, Role: FOLLOWER
I20260927 22:10:05.725276 10234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a2b62863059419babc733f544e21401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a2b62863059419babc733f544e21401" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41917 } }
I20260927 22:10:05.725587 10234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a2b62863059419babc733f544e21401 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:10:05.725733 10234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a2b62863059419babc733f544e21401 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:10:05.725930 10234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a2b62863059419babc733f544e21401 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:10:05.728212 10234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a2b62863059419babc733f544e21401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a2b62863059419babc733f544e21401" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41917 } }
I20260927 22:10:05.728737 10234 leader_election.cc:302] T 00000000000000000000000000000000 P 6a2b62863059419babc733f544e21401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a2b62863059419babc733f544e21401; no voters:
I20260927 22:10:05.729312 10234 leader_election.cc:288] T 00000000000000000000000000000000 P 6a2b62863059419babc733f544e21401 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:10:05.729454 10239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a2b62863059419babc733f544e21401 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:10:05.730234 10239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a2b62863059419babc733f544e21401 [term 1 LEADER]: Becoming Leader. State: Replica: 6a2b62863059419babc733f544e21401, State: Running, Role: LEADER
I20260927 22:10:05.730870 10239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a2b62863059419babc733f544e21401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a2b62863059419babc733f544e21401" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41917 } }
I20260927 22:10:05.731593 10234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a2b62863059419babc733f544e21401 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:10:05.733893 10241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a2b62863059419babc733f544e21401 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a2b62863059419babc733f544e21401. Latest consensus state: current_term: 1 leader_uuid: "6a2b62863059419babc733f544e21401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a2b62863059419babc733f544e21401" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41917 } } }
I20260927 22:10:05.734755 10241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a2b62863059419babc733f544e21401 [sys.catalog]: This master's current role is: LEADER
I20260927 22:10:05.734460 10240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a2b62863059419babc733f544e21401 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a2b62863059419babc733f544e21401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a2b62863059419babc733f544e21401" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41917 } } }
I20260927 22:10:05.735122 10240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a2b62863059419babc733f544e21401 [sys.catalog]: This master's current role is: LEADER
I20260927 22:10:05.744061 10246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:10:05.750782 10246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:10:05.759624 10246 catalog_manager.cc:1347] Generated new cluster ID: 985ede7229ce44138ac233e25d9afa1a
I20260927 22:10:05.759747 10246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:10:05.774020 10246 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:10:05.774473 10246 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:10:05.779857 10246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a2b62863059419babc733f544e21401: Generated new TSK 0
I20260927 22:10:05.780684 10246 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:10:05.795574 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41917
--builtin_ntp_servers=127.0.97.20:33571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 22:10:06.146124 10258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:10:06.146411 10258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:10:06.146564 10258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:10:06.153957 10258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:10:06.154069 10258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:10:06.154247 10258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:10:06.163673 10258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:10:06.165009 10258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:10:06.166952 10258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:10:06.181262 10263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:06.181317 10266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:06.181304 10264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:10:06.182577 10258 server_base.cc:1034] running on GCE node
I20260927 22:10:06.183271 10258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:10:06.184227 10258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:10:06.185449 10258 hybrid_clock.cc:648] HybridClock initialized: now 1790547006185390 us; error 60 us; skew 500 ppm
I20260927 22:10:06.185770 10258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:10:06.188009 10258 webserver.cc:460] Webserver started at http://127.0.97.1:44967/ using document root <none> and password file <none>
I20260927 22:10:06.188709 10258 fs_manager.cc:359] Metadata directory not provided
I20260927 22:10:06.188840 10258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:10:06.189163 10258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:10:06.193548 10258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "4c39f8e48bed4fceaa065aca22a1b50f"
format_stamp: "Formatted at 2026-09-27 22:10:06 on dist-test-slave-mp03"
server_key: "ec6a2ea6e252337b1d9114d328a9dcc3"
server_key_iv: "86970c63735e446daa62e69c94ed1091"
server_key_version: "encryptionkey@0"
I20260927 22:10:06.194353 10258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4c39f8e48bed4fceaa065aca22a1b50f"
format_stamp: "Formatted at 2026-09-27 22:10:06 on dist-test-slave-mp03"
server_key: "ec6a2ea6e252337b1d9114d328a9dcc3"
server_key_iv: "86970c63735e446daa62e69c94ed1091"
server_key_version: "encryptionkey@0"
I20260927 22:10:06.199802 10258 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260927 22:10:06.203474 10272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:06.205011 10258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:10:06.205178 10258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "4c39f8e48bed4fceaa065aca22a1b50f"
format_stamp: "Formatted at 2026-09-27 22:10:06 on dist-test-slave-mp03"
server_key: "ec6a2ea6e252337b1d9114d328a9dcc3"
server_key_iv: "86970c63735e446daa62e69c94ed1091"
server_key_version: "encryptionkey@0"
I20260927 22:10:06.205431 10258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:10:06.211247 10258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:10:06.212205 10258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:10:06.212513 10258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:10:06.213478 10258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:10:06.214977 10258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:10:06.215101 10258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:06.215265 10258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:10:06.215359 10258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:06.247786 10258 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33955
I20260927 22:10:06.247826 10384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33955 every 8 connection(s)
I20260927 22:10:06.249240 10258 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:10:06.252710 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 10258
I20260927 22:10:06.252936 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:10:06.253727 388 external_mini_cluster.cc:1428] Setting key c640048cc878195137bb3ef90283f6e9
I20260927 22:10:06.256971 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41917
--builtin_ntp_servers=127.0.97.20:33571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 22:10:06.258673 10387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:06.251855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46279 (local address 127.0.97.62:41917)
0927 22:10:06.252316 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:06.252322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:06.253882 (+ 1560us) server_negotiation.cc:407] Connection header received
0927 22:10:06.256058 (+ 2176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:06.256062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:06.256111 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:06.256200 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:06.257878 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:06.257883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:06.257886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:10:06.257902 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:06.257944 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:10:06.257956 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:10:06.258064 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:06.258314 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 22:10:06.258366 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":214,"threads_started":1}
I20260927 22:10:06.259961 10388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:06.252528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41917 (local address 127.0.97.1:46279)
0927 22:10:06.253693 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0927 22:10:06.253757 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 22:10:06.255750 (+ 1993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:10:06.256471 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:10:06.256484 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:10:06.256624 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:10:06.257491 (+ 867us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:10:06.257496 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:10:06.257550 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:10:06.257671 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:10:06.258068 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:10:06.258081 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:10:06.258100 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 22:10:06.258392 (+ 292us) client_negotiation.cc:240] Negotiation successful
0927 22:10:06.258709 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":317,"threads_started":1}
I20260927 22:10:06.261616 10385 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41917
I20260927 22:10:06.261916 10385 heartbeater.cc:464] Registering TS with master...
I20260927 22:10:06.262529 10385 heartbeater.cc:511] Master 127.0.97.62:41917 requested a full tablet report, sending...
I20260927 22:10:06.264777 10199 ts_manager.cc:194] Registered new tserver with Master: 4c39f8e48bed4fceaa065aca22a1b50f (127.0.97.1:33955)
W20260927 22:10:06.600260 10389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:10:06.600627 10389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:10:06.600835 10389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:10:06.608449 10389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:10:06.608618 10389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:10:06.608875 10389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:10:06.618424 10389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:10:06.619846 10389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:10:06.621695 10389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:10:06.636032 10397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:06.636097 10394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:06.636121 10395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:10:06.636729 10389 server_base.cc:1034] running on GCE node
I20260927 22:10:06.637362 10389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:10:06.638304 10389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:10:06.639515 10389 hybrid_clock.cc:648] HybridClock initialized: now 1790547006639447 us; error 61 us; skew 500 ppm
I20260927 22:10:06.639846 10389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:10:06.642225 10389 webserver.cc:460] Webserver started at http://127.0.97.2:43047/ using document root <none> and password file <none>
I20260927 22:10:06.643283 10389 fs_manager.cc:359] Metadata directory not provided
I20260927 22:10:06.643424 10389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:10:06.643810 10389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:10:06.646790 10389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "2ff11c3da58841599a8f73e01bb309b5"
format_stamp: "Formatted at 2026-09-27 22:10:06 on dist-test-slave-mp03"
server_key: "daee50e53692b78f2f34c0436f6dd05a"
server_key_iv: "73edeb255ab22b903c400a7c00e14bc1"
server_key_version: "encryptionkey@0"
I20260927 22:10:06.647586 10389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2ff11c3da58841599a8f73e01bb309b5"
format_stamp: "Formatted at 2026-09-27 22:10:06 on dist-test-slave-mp03"
server_key: "daee50e53692b78f2f34c0436f6dd05a"
server_key_iv: "73edeb255ab22b903c400a7c00e14bc1"
server_key_version: "encryptionkey@0"
I20260927 22:10:06.652691 10389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:10:06.656226 10403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:06.657667 10389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 22:10:06.657871 10389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "2ff11c3da58841599a8f73e01bb309b5"
format_stamp: "Formatted at 2026-09-27 22:10:06 on dist-test-slave-mp03"
server_key: "daee50e53692b78f2f34c0436f6dd05a"
server_key_iv: "73edeb255ab22b903c400a7c00e14bc1"
server_key_version: "encryptionkey@0"
I20260927 22:10:06.658138 10389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:10:06.664067 10389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:10:06.664974 10389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:10:06.665252 10389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:10:06.666210 10389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:10:06.667786 10389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:10:06.667920 10389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:06.668066 10389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:10:06.668164 10389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:06.703003 10389 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41419
I20260927 22:10:06.703092 10515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41419 every 8 connection(s)
I20260927 22:10:06.704337 10389 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:10:06.711162 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 10389
I20260927 22:10:06.711364 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:10:06.711930 388 external_mini_cluster.cc:1428] Setting key f0c47acf1cb89da5051eea694547fa70
I20260927 22:10:06.713928 10387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:06.707024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49123 (local address 127.0.97.62:41917)
0927 22:10:06.707228 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:06.707234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:06.708740 (+ 1506us) server_negotiation.cc:407] Connection header received
0927 22:10:06.710454 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:06.710459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:06.710511 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:06.710672 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:06.712250 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:06.712256 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:06.712259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:10:06.712278 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:06.712376 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:10:06.712393 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:10:06.712509 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:06.713561 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0927 22:10:06.713642 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:10:06.714350 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:06.707546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41917 (local address 127.0.97.2:49123)
0927 22:10:06.708575 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0927 22:10:06.708618 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:10:06.710204 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:10:06.710927 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:10:06.710977 (+ 50us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:10:06.711113 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:10:06.711866 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:10:06.711870 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:10:06.711920 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:10:06.712056 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:10:06.712726 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:10:06.712736 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:10:06.712744 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:10:06.713026 (+ 282us) client_negotiation.cc:240] Negotiation successful
0927 22:10:06.713226 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":309,"threads_started":1}
I20260927 22:10:06.715030 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41917
--builtin_ntp_servers=127.0.97.20:33571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 22:10:06.715819 10516 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41917
I20260927 22:10:06.716120 10516 heartbeater.cc:464] Registering TS with master...
I20260927 22:10:06.716780 10516 heartbeater.cc:511] Master 127.0.97.62:41917 requested a full tablet report, sending...
I20260927 22:10:06.718289 10199 ts_manager.cc:194] Registered new tserver with Master: 2ff11c3da58841599a8f73e01bb309b5 (127.0.97.2:41419)
W20260927 22:10:07.046149 10519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:10:07.046521 10519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:10:07.046692 10519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:10:07.053893 10519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:10:07.053993 10519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:10:07.054104 10519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:10:07.063397 10519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:10:07.064643 10519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:10:07.066463 10519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:10:07.081465 10525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:07.081672 10524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:07.082429 10527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:10:07.084497 10519 server_base.cc:1034] running on GCE node
I20260927 22:10:07.085244 10519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:10:07.086221 10519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:10:07.087482 10519 hybrid_clock.cc:648] HybridClock initialized: now 1790547007087431 us; error 64 us; skew 500 ppm
I20260927 22:10:07.087857 10519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:10:07.090209 10519 webserver.cc:460] Webserver started at http://127.0.97.3:36303/ using document root <none> and password file <none>
I20260927 22:10:07.090996 10519 fs_manager.cc:359] Metadata directory not provided
I20260927 22:10:07.091127 10519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:10:07.091431 10519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:10:07.094203 10519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "709c626692b54f0798586b5aba9c9c3c"
format_stamp: "Formatted at 2026-09-27 22:10:07 on dist-test-slave-mp03"
server_key: "fd635db4d2eb0e59173f818d1bfab88c"
server_key_iv: "671cb40ad1976562a3123301b48584b0"
server_key_version: "encryptionkey@0"
I20260927 22:10:07.095005 10519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "709c626692b54f0798586b5aba9c9c3c"
format_stamp: "Formatted at 2026-09-27 22:10:07 on dist-test-slave-mp03"
server_key: "fd635db4d2eb0e59173f818d1bfab88c"
server_key_iv: "671cb40ad1976562a3123301b48584b0"
server_key_version: "encryptionkey@0"
I20260927 22:10:07.100274 10519 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:10:07.103653 10533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:07.104960 10519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:10:07.105130 10519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "709c626692b54f0798586b5aba9c9c3c"
format_stamp: "Formatted at 2026-09-27 22:10:07 on dist-test-slave-mp03"
server_key: "fd635db4d2eb0e59173f818d1bfab88c"
server_key_iv: "671cb40ad1976562a3123301b48584b0"
server_key_version: "encryptionkey@0"
I20260927 22:10:07.105377 10519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:10:07.111248 10519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:10:07.112012 10519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:10:07.112273 10519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:10:07.113130 10519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:10:07.114485 10519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:10:07.114604 10519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:07.114743 10519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:10:07.114835 10519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:07.147389 10519 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41837
I20260927 22:10:07.147442 10645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41837 every 8 connection(s)
I20260927 22:10:07.148684 10519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:10:07.157203 10387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:07.151380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35339 (local address 127.0.97.62:41917)
0927 22:10:07.151556 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:07.151560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:07.153056 (+ 1496us) server_negotiation.cc:407] Connection header received
0927 22:10:07.154876 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:07.154881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:07.154936 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:07.155037 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:07.156506 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:07.156510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:07.156513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:10:07.156529 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:07.156559 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:10:07.156570 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:10:07.156664 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:07.156941 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 22:10:07.156996 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:10:07.158124 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:07.151932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41917 (local address 127.0.97.3:35339)
0927 22:10:07.152876 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0927 22:10:07.152917 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:10:07.154518 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:10:07.155342 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:10:07.155350 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:10:07.155456 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:10:07.156209 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:10:07.156212 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:10:07.156259 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:10:07.156345 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:10:07.156692 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:10:07.156702 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:10:07.156709 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:10:07.156953 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 22:10:07.157160 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":319,"threads_started":1}
I20260927 22:10:07.158707 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 10519
I20260927 22:10:07.158902 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:10:07.159562 388 external_mini_cluster.cc:1428] Setting key d749779ef8c124733d15aba731d092a6
I20260927 22:10:07.160928 10646 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41917
I20260927 22:10:07.161260 10646 heartbeater.cc:464] Registering TS with master...
I20260927 22:10:07.162011 10646 heartbeater.cc:511] Master 127.0.97.62:41917 requested a full tablet report, sending...
I20260927 22:10:07.163476 10199 ts_manager.cc:194] Registered new tserver with Master: 709c626692b54f0798586b5aba9c9c3c (127.0.97.3:41837)
I20260927 22:10:07.175276 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 22:10:07.181478 10387 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:10:07.179896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50070 (local address 127.0.97.62:41917)
0927 22:10:07.180069 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:07.180073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:07.180678 (+ 605us) server_negotiation.cc:407] Connection header received
0927 22:10:07.180848 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:07.180852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:07.180901 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:07.180988 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:07.181262 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50070: BlockingRecv error: recv got EOF from 127.0.0.1:50070 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:10:07.182103 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 10258
I20260927 22:10:07.185916 10655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:07.183148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43552 (local address 127.0.97.1:33955)
0927 22:10:07.183684 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:07.183691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:07.183744 (+ 53us) server_negotiation.cc:407] Connection header received
0927 22:10:07.183804 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:07.183810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:07.184047 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:07.184177 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:07.185164 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:07.185170 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:07.185183 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:10:07.185202 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:07.185247 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:10:07.185262 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:10:07.185368 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:07.185572 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 22:10:07.185667 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":284,"threads_started":1}
I20260927 22:10:07.187234 10380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:10:07.268774 10385 heartbeater.cc:503] Master 127.0.97.62:41917 was elected leader, sending a full tablet report...
I20260927 22:10:07.270464 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 10258
I20260927 22:10:07.287307 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 10389
I20260927 22:10:07.290998 10657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:07.288279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47316 (local address 127.0.97.2:41419)
0927 22:10:07.288760 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:07.288783 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:07.288808 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:10:07.288869 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:07.288875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:07.289096 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:07.289221 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:07.290227 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:07.290233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:07.290245 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:10:07.290258 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:07.290300 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:10:07.290314 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:10:07.290415 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:07.290546 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 22:10:07.290697 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":229,"threads_started":1}
I20260927 22:10:07.292322 10511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:10:07.376859 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 10389
I20260927 22:10:07.392421 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 10519
I20260927 22:10:07.396225 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:07.393452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56432 (local address 127.0.97.3:41837)
0927 22:10:07.394064 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:07.394071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:07.394099 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:10:07.394202 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:07.394209 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:07.394422 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:07.394546 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:07.395616 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:07.395623 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:07.395634 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:10:07.395649 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:07.395686 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:10:07.395700 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:10:07.395824 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:07.395874 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 22:10:07.395948 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":322,"threads_started":1}
I20260927 22:10:07.397446 10641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:10:07.480432 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 10519
I20260927 22:10:07.496333 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 10168
I20260927 22:10:07.497615 10229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:10:07.601954 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 10168
2026-09-27T22:10:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2414 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:10:07 dist-test-slave-mp03 krb5kdc[10664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:10:07 dist-test-slave-mp03 krb5kdc[10664](info): set up 2 sockets
Sep 27 22:10:07 dist-test-slave-mp03 krb5kdc[10664](info): commencing operation
krb5kdc: starting...
W20260927 22:10:08.707623 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:10:08 dist-test-slave-mp03 krb5kdc[10664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:10:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:10:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:10:08.947468 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39691
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:10:09.282851 10677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:10:09.283143 10677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:10:09.283314 10677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:10:09.290786 10677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:10:09.290900 10677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:10:09.291009 10677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:10:09.291115 10677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:10:09.291201 10677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:10:09.301584 10677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:10:09.303223 10677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:10:09.304563 10677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:10:09.316300 10685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:09.316391 10682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:09.316381 10683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:10:09.317152 10677 server_base.cc:1034] running on GCE node
I20260927 22:10:09.317975 10677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:10:09.319535 10677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:10:09.320766 10677 hybrid_clock.cc:648] HybridClock initialized: now 1790547009320707 us; error 64 us; skew 500 ppm
I20260927 22:10:09.321110 10677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:10:09 dist-test-slave-mp03 krb5kdc[10664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:10:09.330195 10677 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:10:09.331792 10677 webserver.cc:460] Webserver started at http://127.0.97.62:39465/ using document root <none> and password file <none>
I20260927 22:10:09.332474 10677 fs_manager.cc:359] Metadata directory not provided
I20260927 22:10:09.332581 10677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:10:09.332826 10677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:10:09.335861 10677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "834bcc2e893a4d6888d7ad8c80e95069"
format_stamp: "Formatted at 2026-09-27 22:10:09 on dist-test-slave-mp03"
server_key: "02210520ec35926691c6c34fd3109577"
server_key_iv: "6f0af79a571e5f887b55f893f9e93e0e"
server_key_version: "encryptionkey@0"
I20260927 22:10:09.336577 10677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "834bcc2e893a4d6888d7ad8c80e95069"
format_stamp: "Formatted at 2026-09-27 22:10:09 on dist-test-slave-mp03"
server_key: "02210520ec35926691c6c34fd3109577"
server_key_iv: "6f0af79a571e5f887b55f893f9e93e0e"
server_key_version: "encryptionkey@0"
I20260927 22:10:09.342156 10677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 22:10:09.345687 10692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:09.347136 10677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:10:09.347314 10677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "834bcc2e893a4d6888d7ad8c80e95069"
format_stamp: "Formatted at 2026-09-27 22:10:09 on dist-test-slave-mp03"
server_key: "02210520ec35926691c6c34fd3109577"
server_key_iv: "6f0af79a571e5f887b55f893f9e93e0e"
server_key_version: "encryptionkey@0"
I20260927 22:10:09.347505 10677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:10:09.365252 10677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:10:09.369650 10677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:10:09.369928 10677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:10:09.388888 10677 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39691
I20260927 22:10:09.388911 10743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39691 every 8 connection(s)
I20260927 22:10:09.390225 10677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:10:09.394232 10744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:10:09.400460 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 10677
I20260927 22:10:09.400707 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:10:09.401521 388 external_mini_cluster.cc:1428] Setting key 280b2f0ac61fb84cbbece965f93abf5d
I20260927 22:10:09.405210 10744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 834bcc2e893a4d6888d7ad8c80e95069: Bootstrap starting.
I20260927 22:10:09.408241 10744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 834bcc2e893a4d6888d7ad8c80e95069: Neither blocks nor log segments found. Creating new log.
I20260927 22:10:09.409279 10744 log.cc:824] T 00000000000000000000000000000000 P 834bcc2e893a4d6888d7ad8c80e95069: Log is configured to *not* fsync() on all Append() calls
I20260927 22:10:09.412592 10744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 834bcc2e893a4d6888d7ad8c80e95069: No bootstrap required, opened a new log
Sep 27 22:10:09 dist-test-slave-mp03 krb5kdc[10664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:10:09.423449 10744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 834bcc2e893a4d6888d7ad8c80e95069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834bcc2e893a4d6888d7ad8c80e95069" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } }
I20260927 22:10:09.423960 10744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 834bcc2e893a4d6888d7ad8c80e95069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:10:09.424070 10744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 834bcc2e893a4d6888d7ad8c80e95069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 834bcc2e893a4d6888d7ad8c80e95069, State: Initialized, Role: FOLLOWER
I20260927 22:10:09.424777 10744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 834bcc2e893a4d6888d7ad8c80e95069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834bcc2e893a4d6888d7ad8c80e95069" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } }
I20260927 22:10:09.425078 10744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 834bcc2e893a4d6888d7ad8c80e95069 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:10:09.425242 10744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 834bcc2e893a4d6888d7ad8c80e95069 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:10:09.425426 10744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 834bcc2e893a4d6888d7ad8c80e95069 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:10:09.427211 10747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:09.403649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60482 (local address 127.0.97.62:39691)
0927 22:10:09.404375 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:09.404387 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:09.404427 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:10:09.406257 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:09.406279 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:09.406701 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:09.407081 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:09.408596 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:09.410280 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:09.412105 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:09.412840 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:09.412979 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:09.417687 (+ 4708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:09.417702 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:09.417730 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:10:09.417793 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:09.422125 (+ 4332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:10:09.423026 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:10:09.423034 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:10:09.423043 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:10:09.423252 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:10:09.423733 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:10:09.423737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:10:09.423740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:10:09.424487 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:10:09.424786 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:09.425183 (+ 397us) server_negotiation.cc:299] Negotiation successful
0927 22:10:09.425681 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":335,"threads_started":1}
I20260927 22:10:09.428840 10744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 834bcc2e893a4d6888d7ad8c80e95069 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834bcc2e893a4d6888d7ad8c80e95069" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } }
I20260927 22:10:09.429347 10744 leader_election.cc:302] T 00000000000000000000000000000000 P 834bcc2e893a4d6888d7ad8c80e95069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 834bcc2e893a4d6888d7ad8c80e95069; no voters:
I20260927 22:10:09.429960 10744 leader_election.cc:288] T 00000000000000000000000000000000 P 834bcc2e893a4d6888d7ad8c80e95069 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:10:09.430115 10749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 834bcc2e893a4d6888d7ad8c80e95069 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:10:09.431226 10749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 834bcc2e893a4d6888d7ad8c80e95069 [term 1 LEADER]: Becoming Leader. State: Replica: 834bcc2e893a4d6888d7ad8c80e95069, State: Running, Role: LEADER
I20260927 22:10:09.431710 10749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 834bcc2e893a4d6888d7ad8c80e95069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834bcc2e893a4d6888d7ad8c80e95069" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } }
I20260927 22:10:09.432035 10744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 834bcc2e893a4d6888d7ad8c80e95069 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:10:09.434360 10751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 834bcc2e893a4d6888d7ad8c80e95069 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 834bcc2e893a4d6888d7ad8c80e95069. Latest consensus state: current_term: 1 leader_uuid: "834bcc2e893a4d6888d7ad8c80e95069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834bcc2e893a4d6888d7ad8c80e95069" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } } }
I20260927 22:10:09.434629 10750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 834bcc2e893a4d6888d7ad8c80e95069 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "834bcc2e893a4d6888d7ad8c80e95069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834bcc2e893a4d6888d7ad8c80e95069" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } } }
I20260927 22:10:09.434909 10750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 834bcc2e893a4d6888d7ad8c80e95069 [sys.catalog]: This master's current role is: LEADER
I20260927 22:10:09.434818 10751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 834bcc2e893a4d6888d7ad8c80e95069 [sys.catalog]: This master's current role is: LEADER
I20260927 22:10:09.436866 10758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:10:09.442253 10758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:10:09.451506 10758 catalog_manager.cc:1347] Generated new cluster ID: 42f40903f60e41eb9f00f388cad4e385
I20260927 22:10:09.451663 10758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:10:09.468606 10758 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:10:09.470046 10758 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:10:09.478132 10758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 834bcc2e893a4d6888d7ad8c80e95069: Generated new TSK 0
I20260927 22:10:09.479275 10758 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:10:09.550285 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39691
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:10:09.905301 10770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:10:09.905668 10770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:10:09.905846 10770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:10:09.913362 10770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:10:09.913534 10770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:10:09.913705 10770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:10:09.923825 10770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:10:09.925251 10770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:10:09.926858 10770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:10:09.941170 10776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:09.941102 10775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:09.941146 10778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:10:09.942058 10770 server_base.cc:1034] running on GCE node
I20260927 22:10:09.942767 10770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:10:09.943794 10770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:10:09.945029 10770 hybrid_clock.cc:648] HybridClock initialized: now 1790547009944981 us; error 45 us; skew 500 ppm
I20260927 22:10:09.945609 10770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:10:09 dist-test-slave-mp03 krb5kdc[10664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:10:09.949046 10770 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:10:09.950752 10770 webserver.cc:460] Webserver started at http://127.0.97.1:42715/ using document root <none> and password file <none>
I20260927 22:10:09.951537 10770 fs_manager.cc:359] Metadata directory not provided
I20260927 22:10:09.951685 10770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:10:09.952041 10770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:10:09.955089 10770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "cf4342f2259649b69d3d545cf51f5969"
format_stamp: "Formatted at 2026-09-27 22:10:09 on dist-test-slave-mp03"
server_key: "a88e6e212fe24a2c196287ffa2921486"
server_key_iv: "af4b026afe46d2a82ae8d8030442cd65"
server_key_version: "encryptionkey@0"
I20260927 22:10:09.955870 10770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "cf4342f2259649b69d3d545cf51f5969"
format_stamp: "Formatted at 2026-09-27 22:10:09 on dist-test-slave-mp03"
server_key: "a88e6e212fe24a2c196287ffa2921486"
server_key_iv: "af4b026afe46d2a82ae8d8030442cd65"
server_key_version: "encryptionkey@0"
I20260927 22:10:09.961236 10770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:10:09.964951 10785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:09.966483 10770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:10:09.966696 10770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "cf4342f2259649b69d3d545cf51f5969"
format_stamp: "Formatted at 2026-09-27 22:10:09 on dist-test-slave-mp03"
server_key: "a88e6e212fe24a2c196287ffa2921486"
server_key_iv: "af4b026afe46d2a82ae8d8030442cd65"
server_key_version: "encryptionkey@0"
I20260927 22:10:09.966907 10770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:10:09.994221 10770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:10:09.998725 10770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:10:09.999035 10770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:10:10.000061 10770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:10:10.001502 10770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:10:10.001612 10770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:10.001761 10770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:10:10.001834 10770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:10.035845 10770 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41561
I20260927 22:10:10.035885 10897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41561 every 8 connection(s)
I20260927 22:10:10.037245 10770 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:10:10.037865 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 10770
I20260927 22:10:10.038091 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:10:10.038846 388 external_mini_cluster.cc:1428] Setting key 82a4440b05c860063348add588b83eac
Sep 27 22:10:10 dist-test-slave-mp03 krb5kdc[10664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:10:10.060973 10900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:10.040205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51899 (local address 127.0.97.62:39691)
0927 22:10:10.040676 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:10.040681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:10.041916 (+ 1235us) server_negotiation.cc:407] Connection header received
0927 22:10:10.043960 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:10.043966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:10.044056 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:10.044181 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:10.046239 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:10.047457 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:10.049067 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:10.049843 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:10.049994 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:10.053692 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:10.053701 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:10.053709 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:10:10.053756 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:10.057416 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:10:10.058191 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:10:10.058200 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:10:10.058206 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:10:10.058359 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:10:10.058857 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:10:10.058865 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:10:10.058871 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:10:10.059255 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:10:10.059426 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:10.060210 (+ 784us) server_negotiation.cc:299] Negotiation successful
0927 22:10:10.060482 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":251,"threads_started":1}
I20260927 22:10:10.061647 10901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:10.040928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39691 (local address 127.0.97.1:51899)
0927 22:10:10.041747 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0927 22:10:10.041793 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:10:10.043669 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:10:10.044441 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:10:10.044450 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:10:10.045038 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:10:10.046023 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:10.046071 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:10.047633 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:10.047637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:10.048894 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:10.048906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:10.050069 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:10.050075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:10.050280 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:10.051095 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:10:10.051130 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:10:10.053450 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:10:10.057583 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:10:10.057590 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:10:10.057604 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:10:10.058060 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:10:10.058478 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:10:10.058483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:10:10.058487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:10:10.058731 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:10:10.059403 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:10:10.059409 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:10:10.059782 (+ 373us) client_negotiation.cc:769] Sending connection context
0927 22:10:10.060019 (+ 237us) client_negotiation.cc:240] Negotiation successful
0927 22:10:10.060421 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":340,"threads_started":1}
I20260927 22:10:10.063311 10898 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39691
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:10:10.063663 10898 heartbeater.cc:464] Registering TS with master...
I20260927 22:10:10.064379 10898 heartbeater.cc:511] Master 127.0.97.62:39691 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 22:10:10.066783 10709 ts_manager.cc:194] Registered new tserver with Master: cf4342f2259649b69d3d545cf51f5969 (127.0.97.1:41561)
I20260927 22:10:10.068286 10709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:10:10.093833 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39691
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:10:10.435770 10904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:10:10.436069 10904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:10:10.436259 10904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:10:10.443980 10904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:10:10.444101 10904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:10:10.444301 10904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:10:10.454257 10904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:10:10.455715 10904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:10:10.457086 10904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:10:10.471958 10910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:10.471980 10909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:10.471980 10912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:10:10.473086 10904 server_base.cc:1034] running on GCE node
I20260927 22:10:10.473762 10904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:10:10.474845 10904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:10:10.476101 10904 hybrid_clock.cc:648] HybridClock initialized: now 1790547010476034 us; error 58 us; skew 500 ppm
I20260927 22:10:10.476496 10904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:10:10 dist-test-slave-mp03 krb5kdc[10664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:10:10.480026 10904 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:10:10.481551 10904 webserver.cc:460] Webserver started at http://127.0.97.2:38531/ using document root <none> and password file <none>
I20260927 22:10:10.482281 10904 fs_manager.cc:359] Metadata directory not provided
I20260927 22:10:10.482429 10904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:10:10.482808 10904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:10:10.485873 10904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "4214deac0dbd4b5097029d85533ebba3"
format_stamp: "Formatted at 2026-09-27 22:10:10 on dist-test-slave-mp03"
server_key: "ea5453a78d10cc8afb14798827a0ebd8"
server_key_iv: "bf62c17bda1541bd8c0aa2b8c2d1697d"
server_key_version: "encryptionkey@0"
I20260927 22:10:10.486739 10904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4214deac0dbd4b5097029d85533ebba3"
format_stamp: "Formatted at 2026-09-27 22:10:10 on dist-test-slave-mp03"
server_key: "ea5453a78d10cc8afb14798827a0ebd8"
server_key_iv: "bf62c17bda1541bd8c0aa2b8c2d1697d"
server_key_version: "encryptionkey@0"
I20260927 22:10:10.492005 10904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 22:10:10.495769 10919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:10.497251 10904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:10:10.497454 10904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "4214deac0dbd4b5097029d85533ebba3"
format_stamp: "Formatted at 2026-09-27 22:10:10 on dist-test-slave-mp03"
server_key: "ea5453a78d10cc8afb14798827a0ebd8"
server_key_iv: "bf62c17bda1541bd8c0aa2b8c2d1697d"
server_key_version: "encryptionkey@0"
I20260927 22:10:10.497673 10904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:10:10.518848 10904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:10:10.522998 10904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:10:10.523418 10904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:10:10.524413 10904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:10:10.525725 10904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:10:10.525828 10904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:10.526003 10904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:10:10.526077 10904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:10.557705 10904 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43081
I20260927 22:10:10.557863 11031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43081 every 8 connection(s)
I20260927 22:10:10.559232 10904 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:10:10.566983 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 10904
I20260927 22:10:10.567224 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:10:10.567785 388 external_mini_cluster.cc:1428] Setting key c07e798da73ae6a0d13e53a20d8ac1f2
Sep 27 22:10:10 dist-test-slave-mp03 krb5kdc[10664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:10:10.584789 11034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:10.562082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57019 (local address 127.0.97.62:39691)
0927 22:10:10.562579 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:10.562583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:10.563812 (+ 1229us) server_negotiation.cc:407] Connection header received
0927 22:10:10.565882 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:10.565889 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:10.566017 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:10.566146 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:10.568506 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:10.569445 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:10.572202 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:10.572673 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:10.572790 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:10.577543 (+ 4753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:10.577548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:10.577552 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:10:10.577586 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:10.581175 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:10:10.582037 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:10:10.582045 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:10:10.582052 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:10:10.582226 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:10:10.582748 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:10:10.582755 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:10:10.582761 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:10:10.583143 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:10:10.583299 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:10.584033 (+ 734us) server_negotiation.cc:299] Negotiation successful
0927 22:10:10.584295 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":262,"threads_started":1}
I20260927 22:10:10.585548 11035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:10.562822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39691 (local address 127.0.97.2:57019)
0927 22:10:10.563610 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0927 22:10:10.563651 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:10:10.565559 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:10:10.566415 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:10:10.566424 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:10:10.567320 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:10:10.568304 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:10.568332 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:10.569615 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:10.569623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:10.572008 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:10.572030 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:10.572939 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:10.572949 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:10.573311 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:10.574104 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:10:10.574148 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:10:10.577245 (+ 3097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:10:10.581372 (+ 4127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:10:10.581380 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:10:10.581396 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:10:10.581878 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:10:10.582353 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:10:10.582356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:10:10.582358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:10:10.582546 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:10:10.583275 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:10:10.583282 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:10:10.583632 (+ 350us) client_negotiation.cc:769] Sending connection context
0927 22:10:10.583882 (+ 250us) client_negotiation.cc:240] Negotiation successful
0927 22:10:10.584310 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":330,"threads_started":1}
I20260927 22:10:10.587610 11032 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39691
I20260927 22:10:10.587965 11032 heartbeater.cc:464] Registering TS with master...
I20260927 22:10:10.588781 11032 heartbeater.cc:511] Master 127.0.97.62:39691 requested a full tablet report, sending...
I20260927 22:10:10.590437 10709 ts_manager.cc:194] Registered new tserver with Master: 4214deac0dbd4b5097029d85533ebba3 (127.0.97.2:43081)
I20260927 22:10:10.591367 10709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57019
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:10:10.626442 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39691
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:10:10.957852 11038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:10:10.958161 11038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:10:10.958325 11038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:10:10.965569 11038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:10:10.965732 11038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:10:10.965929 11038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:10:10.976475 11038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:10:10.978039 11038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:10:10.979537 11038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:10:10.995297 11043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:10.995257 11046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:10.995316 11044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:10:10.996016 11038 server_base.cc:1034] running on GCE node
I20260927 22:10:10.996621 11038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:10:10.997592 11038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:10:10.998806 11038 hybrid_clock.cc:648] HybridClock initialized: now 1790547010998755 us; error 40 us; skew 500 ppm
I20260927 22:10:10.999176 11038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:10:10 dist-test-slave-mp03 krb5kdc[10664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:10:11.002321 11038 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:10:11.003829 11038 webserver.cc:460] Webserver started at http://127.0.97.3:41749/ using document root <none> and password file <none>
I20260927 22:10:11.004531 11038 fs_manager.cc:359] Metadata directory not provided
I20260927 22:10:11.004678 11038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:10:11.004984 11038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:10:11.007982 11038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "179737049c8d492b990cf2cc8817e494"
format_stamp: "Formatted at 2026-09-27 22:10:11 on dist-test-slave-mp03"
server_key: "1c0c70cfc713b418e284b9a0f77159a4"
server_key_iv: "dc978a78dd88b210d532796c61652dfe"
server_key_version: "encryptionkey@0"
I20260927 22:10:11.008764 11038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "179737049c8d492b990cf2cc8817e494"
format_stamp: "Formatted at 2026-09-27 22:10:11 on dist-test-slave-mp03"
server_key: "1c0c70cfc713b418e284b9a0f77159a4"
server_key_iv: "dc978a78dd88b210d532796c61652dfe"
server_key_version: "encryptionkey@0"
I20260927 22:10:11.013973 11038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 22:10:11.017606 11053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:11.019111 11038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:10:11.019351 11038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "179737049c8d492b990cf2cc8817e494"
format_stamp: "Formatted at 2026-09-27 22:10:11 on dist-test-slave-mp03"
server_key: "1c0c70cfc713b418e284b9a0f77159a4"
server_key_iv: "dc978a78dd88b210d532796c61652dfe"
server_key_version: "encryptionkey@0"
I20260927 22:10:11.019567 11038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:10:11.035148 11038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:10:11.039522 11038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:10:11.039821 11038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:10:11.040863 11038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:10:11.042246 11038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:10:11.042342 11038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:11.042480 11038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:10:11.042557 11038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:11.071374 10898 heartbeater.cc:503] Master 127.0.97.62:39691 was elected leader, sending a full tablet report...
I20260927 22:10:11.077219 11038 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41231
I20260927 22:10:11.077242 11165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41231 every 8 connection(s)
I20260927 22:10:11.078646 11038 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:10:11.079187 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 11038
I20260927 22:10:11.079401 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:10:11.081045 388 external_mini_cluster.cc:1428] Setting key 36265ae5ed399e32c8ae938add5b738e
Sep 27 22:10:11 dist-test-slave-mp03 krb5kdc[10664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:10:11.100767 11034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:11.082308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52443 (local address 127.0.97.62:39691)
0927 22:10:11.082533 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:11.082539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:11.083925 (+ 1386us) server_negotiation.cc:407] Connection header received
0927 22:10:11.085892 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:11.085896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:11.086000 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:11.086108 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:11.088044 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.088855 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.090413 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.091005 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.091120 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:11.094500 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:11.094505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:11.094508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:10:11.094539 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:11.097880 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:10:11.098674 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:10:11.098678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:10:11.098680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:10:11.098795 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:10:11.099291 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:10:11.099294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:10:11.099296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:10:11.099556 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:10:11.099658 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:11.100374 (+ 716us) server_negotiation.cc:299] Negotiation successful
0927 22:10:11.100539 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260927 22:10:11.101858 11168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:11.082726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39691 (local address 127.0.97.3:52443)
0927 22:10:11.083790 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0927 22:10:11.083827 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:10:11.085602 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:10:11.086411 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:10:11.086426 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:10:11.087178 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:10:11.087849 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:11.087879 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.088992 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.088996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:11.090256 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:11.090268 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.091223 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.091229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:11.091433 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:11.092123 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:10:11.092147 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:10:11.094304 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:10:11.098024 (+ 3720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:10:11.098032 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:10:11.098048 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:10:11.098507 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:10:11.098940 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:10:11.098944 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:10:11.098946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:10:11.099184 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:10:11.099672 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:10:11.099678 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:10:11.100012 (+ 334us) client_negotiation.cc:769] Sending connection context
0927 22:10:11.100275 (+ 263us) client_negotiation.cc:240] Negotiation successful
0927 22:10:11.100688 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":549,"threads_started":1}
I20260927 22:10:11.103319 11166 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39691
I20260927 22:10:11.103617 11166 heartbeater.cc:464] Registering TS with master...
I20260927 22:10:11.104257 11166 heartbeater.cc:511] Master 127.0.97.62:39691 requested a full tablet report, sending...
I20260927 22:10:11.105626 10709 ts_manager.cc:194] Registered new tserver with Master: 179737049c8d492b990cf2cc8817e494 (127.0.97.3:41231)
I20260927 22:10:11.106405 10709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52443
I20260927 22:10:11.109674 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:10:11.126112 11034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:11.113835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60494 (local address 127.0.97.62:39691)
0927 22:10:11.114011 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:11.114020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:11.114415 (+ 395us) server_negotiation.cc:407] Connection header received
0927 22:10:11.114650 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:11.114655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:11.114727 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:11.114831 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:11.116188 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.116913 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.118136 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.118526 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.118699 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:11.120992 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:11.120998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:11.121002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:10:11.121041 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:11.123632 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:10:11.124274 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:10:11.124281 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:10:11.124285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:10:11.124430 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:10:11.124796 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:10:11.124803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:10:11.124806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:10:11.125173 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:10:11.125411 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:11.125696 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 22:10:11.125871 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:10:11.130174 10709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:10:11.137516 10709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:10:11.174129 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:11.150687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41561 (local address 127.0.0.1:37632)
0927 22:10:11.165703 (+ 15016us) negotiation.cc:107] Waiting for socket to connect
0927 22:10:11.165755 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 22:10:11.165984 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:10:11.166668 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:10:11.166675 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:10:11.166710 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:10:11.167134 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:11.167153 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.169128 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.169132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:11.171726 (+ 2594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:11.171735 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.173040 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.173047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:11.173229 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:11.173283 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 22:10:11.173420 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 22:10:11.173626 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14833,"mutex_wait_us":103,"thread_start_us":13231,"threads_started":1}
I20260927 22:10:11.174652 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:11.150948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43204 (local address 127.0.97.2:43081)
0927 22:10:11.161499 (+ 10551us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:11.161511 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:11.164688 (+ 3177us) server_negotiation.cc:407] Connection header received
0927 22:10:11.165543 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:11.165554 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:11.165900 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:11.166101 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:10:11.167005 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.168808 (+ 1803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.171851 (+ 3043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.173213 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.173325 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:11.173487 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:11.174124 (+ 637us) server_negotiation.cc:299] Negotiation successful
0927 22:10:11.174299 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10362,"thread_start_us":10235,"threads_started":1}
I20260927 22:10:11.175138 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:11.150878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37632 (local address 127.0.97.1:41561)
0927 22:10:11.163561 (+ 12683us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:11.163573 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:11.165870 (+ 2297us) server_negotiation.cc:407] Connection header received
0927 22:10:11.166164 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:11.166172 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:11.166425 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:11.166643 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:10:11.167353 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.168986 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.171888 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.172847 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.172988 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:11.173243 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:11.174618 (+ 1375us) server_negotiation.cc:299] Negotiation successful
0927 22:10:11.174806 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12457,"thread_start_us":12692,"threads_started":1}
I20260927 22:10:11.175177 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:11.154291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43358 (local address 127.0.97.3:41231)
0927 22:10:11.165174 (+ 10883us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:11.165187 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:11.166251 (+ 1064us) server_negotiation.cc:407] Connection header received
0927 22:10:11.166352 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:11.166359 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:11.166528 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:11.166761 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:10:11.167579 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.169093 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.171603 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.173003 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.173129 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:11.173443 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:11.174703 (+ 1260us) server_negotiation.cc:299] Negotiation successful
0927 22:10:11.174877 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10664,"thread_start_us":10483,"threads_started":1}
I20260927 22:10:11.174574 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:11.153679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41231 (local address 127.0.0.1:43358)
0927 22:10:11.166000 (+ 12321us) negotiation.cc:107] Waiting for socket to connect
0927 22:10:11.166019 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:10:11.166127 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:10:11.166892 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:10:11.166903 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:10:11.166927 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:10:11.167404 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:11.167416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.169238 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.169243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:11.171451 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:11.171463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.173624 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.173631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:11.173838 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:11.173946 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 22:10:11.174047 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 22:10:11.174190 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12194,"mutex_wait_us":103,"thread_start_us":11662,"threads_started":1}
I20260927 22:10:11.176812 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:11.150687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43081 (local address 127.0.0.1:43204)
0927 22:10:11.164426 (+ 13739us) negotiation.cc:107] Waiting for socket to connect
0927 22:10:11.164550 (+ 124us) client_negotiation.cc:174] Beginning negotiation
0927 22:10:11.165217 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:10:11.166264 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:10:11.166271 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:10:11.166299 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:10:11.166801 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:11.166820 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.168930 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.168939 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:11.171693 (+ 2754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:11.171709 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.173652 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.173658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:11.173839 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:11.173951 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 22:10:11.174906 (+ 955us) client_negotiation.cc:240] Negotiation successful
0927 22:10:11.175043 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13408,"mutex_wait_us":151,"thread_start_us":15014,"threads_started":1}
I20260927 22:10:11.178846 10833 tablet_service.cc:1467] Processing CreateTablet for tablet 5e3fa6bc6ae84baa81411d2f45ebf840 (DEFAULT_TABLE table=test-table [id=b5a454c48b8f4c16873c31da286272a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:10:11.180166 10833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e3fa6bc6ae84baa81411d2f45ebf840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:10:11.180400 11101 tablet_service.cc:1467] Processing CreateTablet for tablet 5e3fa6bc6ae84baa81411d2f45ebf840 (DEFAULT_TABLE table=test-table [id=b5a454c48b8f4c16873c31da286272a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:10:11.180402 10967 tablet_service.cc:1467] Processing CreateTablet for tablet 5e3fa6bc6ae84baa81411d2f45ebf840 (DEFAULT_TABLE table=test-table [id=b5a454c48b8f4c16873c31da286272a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:10:11.181679 11101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e3fa6bc6ae84baa81411d2f45ebf840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:10:11.181672 10967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e3fa6bc6ae84baa81411d2f45ebf840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:10:11.191381 11184 tablet_bootstrap.cc:492] T 5e3fa6bc6ae84baa81411d2f45ebf840 P cf4342f2259649b69d3d545cf51f5969: Bootstrap starting.
I20260927 22:10:11.194402 11184 tablet_bootstrap.cc:654] T 5e3fa6bc6ae84baa81411d2f45ebf840 P cf4342f2259649b69d3d545cf51f5969: Neither blocks nor log segments found. Creating new log.
I20260927 22:10:11.195564 11184 log.cc:824] T 5e3fa6bc6ae84baa81411d2f45ebf840 P cf4342f2259649b69d3d545cf51f5969: Log is configured to *not* fsync() on all Append() calls
I20260927 22:10:11.196465 11186 tablet_bootstrap.cc:492] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494: Bootstrap starting.
I20260927 22:10:11.196931 11185 tablet_bootstrap.cc:492] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 4214deac0dbd4b5097029d85533ebba3: Bootstrap starting.
I20260927 22:10:11.199190 11184 tablet_bootstrap.cc:492] T 5e3fa6bc6ae84baa81411d2f45ebf840 P cf4342f2259649b69d3d545cf51f5969: No bootstrap required, opened a new log
I20260927 22:10:11.199460 11184 ts_tablet_manager.cc:1397] T 5e3fa6bc6ae84baa81411d2f45ebf840 P cf4342f2259649b69d3d545cf51f5969: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 22:10:11.200474 11185 tablet_bootstrap.cc:654] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 4214deac0dbd4b5097029d85533ebba3: Neither blocks nor log segments found. Creating new log.
I20260927 22:10:11.200706 11186 tablet_bootstrap.cc:654] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494: Neither blocks nor log segments found. Creating new log.
I20260927 22:10:11.201730 11185 log.cc:824] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 4214deac0dbd4b5097029d85533ebba3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:10:11.202263 11186 log.cc:824] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494: Log is configured to *not* fsync() on all Append() calls
I20260927 22:10:11.204605 11185 tablet_bootstrap.cc:492] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 4214deac0dbd4b5097029d85533ebba3: No bootstrap required, opened a new log
I20260927 22:10:11.204862 11185 ts_tablet_manager.cc:1397] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 4214deac0dbd4b5097029d85533ebba3: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 22:10:11.205246 11186 tablet_bootstrap.cc:492] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494: No bootstrap required, opened a new log
I20260927 22:10:11.205494 11186 ts_tablet_manager.cc:1397] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 22:10:11.209669 11184 raft_consensus.cc:348] T 5e3fa6bc6ae84baa81411d2f45ebf840 P cf4342f2259649b69d3d545cf51f5969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4342f2259649b69d3d545cf51f5969" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41561 } } peers { permanent_uuid: "4214deac0dbd4b5097029d85533ebba3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43081 } } peers { permanent_uuid: "179737049c8d492b990cf2cc8817e494" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41231 } }
I20260927 22:10:11.210132 11184 raft_consensus.cc:374] T 5e3fa6bc6ae84baa81411d2f45ebf840 P cf4342f2259649b69d3d545cf51f5969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:10:11.210227 11184 raft_consensus.cc:729] T 5e3fa6bc6ae84baa81411d2f45ebf840 P cf4342f2259649b69d3d545cf51f5969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf4342f2259649b69d3d545cf51f5969, State: Initialized, Role: FOLLOWER
I20260927 22:10:11.210908 11184 consensus_queue.cc:260] T 5e3fa6bc6ae84baa81411d2f45ebf840 P cf4342f2259649b69d3d545cf51f5969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4342f2259649b69d3d545cf51f5969" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41561 } } peers { permanent_uuid: "4214deac0dbd4b5097029d85533ebba3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43081 } } peers { permanent_uuid: "179737049c8d492b990cf2cc8817e494" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41231 } }
I20260927 22:10:11.212704 11184 ts_tablet_manager.cc:1428] T 5e3fa6bc6ae84baa81411d2f45ebf840 P cf4342f2259649b69d3d545cf51f5969: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260927 22:10:11.213624 11185 raft_consensus.cc:348] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 4214deac0dbd4b5097029d85533ebba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4342f2259649b69d3d545cf51f5969" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41561 } } peers { permanent_uuid: "4214deac0dbd4b5097029d85533ebba3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43081 } } peers { permanent_uuid: "179737049c8d492b990cf2cc8817e494" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41231 } }
I20260927 22:10:11.214459 11185 raft_consensus.cc:374] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 4214deac0dbd4b5097029d85533ebba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:10:11.214558 11185 raft_consensus.cc:729] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 4214deac0dbd4b5097029d85533ebba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4214deac0dbd4b5097029d85533ebba3, State: Initialized, Role: FOLLOWER
I20260927 22:10:11.215173 11185 consensus_queue.cc:260] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 4214deac0dbd4b5097029d85533ebba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4342f2259649b69d3d545cf51f5969" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41561 } } peers { permanent_uuid: "4214deac0dbd4b5097029d85533ebba3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43081 } } peers { permanent_uuid: "179737049c8d492b990cf2cc8817e494" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41231 } }
I20260927 22:10:11.215698 11186 raft_consensus.cc:348] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4342f2259649b69d3d545cf51f5969" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41561 } } peers { permanent_uuid: "4214deac0dbd4b5097029d85533ebba3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43081 } } peers { permanent_uuid: "179737049c8d492b990cf2cc8817e494" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41231 } }
I20260927 22:10:11.216274 11032 heartbeater.cc:503] Master 127.0.97.62:39691 was elected leader, sending a full tablet report...
I20260927 22:10:11.216847 11185 ts_tablet_manager.cc:1428] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 4214deac0dbd4b5097029d85533ebba3: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260927 22:10:11.217499 11186 raft_consensus.cc:374] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:10:11.217607 11186 raft_consensus.cc:729] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 179737049c8d492b990cf2cc8817e494, State: Initialized, Role: FOLLOWER
I20260927 22:10:11.218252 11186 consensus_queue.cc:260] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4342f2259649b69d3d545cf51f5969" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41561 } } peers { permanent_uuid: "4214deac0dbd4b5097029d85533ebba3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43081 } } peers { permanent_uuid: "179737049c8d492b990cf2cc8817e494" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41231 } }
I20260927 22:10:11.219439 11166 heartbeater.cc:503] Master 127.0.97.62:39691 was elected leader, sending a full tablet report...
I20260927 22:10:11.220000 11186 ts_tablet_manager.cc:1428] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260927 22:10:11.241760 11192 raft_consensus.cc:482] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:10:11.242161 11192 raft_consensus.cc:504] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4342f2259649b69d3d545cf51f5969" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41561 } } peers { permanent_uuid: "4214deac0dbd4b5097029d85533ebba3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43081 } } peers { permanent_uuid: "179737049c8d492b990cf2cc8817e494" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41231 } }
I20260927 22:10:11.244055 11192 leader_election.cc:288] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf4342f2259649b69d3d545cf51f5969 (127.0.97.1:41561), 4214deac0dbd4b5097029d85533ebba3 (127.0.97.2:43081)
I20260927 22:10:11.250833 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:11.244414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38917 (local address 127.0.97.1:41561)
0927 22:10:11.244669 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:11.244677 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:11.245195 (+ 518us) server_negotiation.cc:407] Connection header received
0927 22:10:11.245286 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:11.245292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:11.245369 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:11.245476 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:10:11.245937 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.246958 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.248700 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.249856 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.249995 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:11.250099 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:11.250372 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 22:10:11.250522 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260927 22:10:11.250833 11168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:11.244442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41561 (local address 127.0.97.3:38917)
0927 22:10:11.244833 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 22:10:11.244851 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:10:11.245034 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:10:11.245521 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:10:11.245530 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:10:11.245562 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:10:11.245783 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:11.245793 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.247101 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.247106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:11.248548 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:11.248559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.250058 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.250064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:11.250214 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:11.250240 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 22:10:11.250328 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 22:10:11.250421 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260}
I20260927 22:10:11.251668 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:11.244672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42523 (local address 127.0.97.2:43081)
0927 22:10:11.244861 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:11.244869 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:11.245478 (+ 609us) server_negotiation.cc:407] Connection header received
0927 22:10:11.245701 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:11.245707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:11.245793 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:11.245891 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:10:11.246484 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.247328 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.249258 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.250386 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.250516 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:11.250624 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:11.251243 (+ 619us) server_negotiation.cc:299] Negotiation successful
0927 22:10:11.251379 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 22:10:11.251698 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:11.244611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43081 (local address 127.0.97.3:42523)
0927 22:10:11.245365 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0927 22:10:11.245383 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:10:11.245533 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:10:11.245925 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:10:11.245930 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:10:11.245955 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:10:11.246324 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:11.246336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.247491 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.247495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:11.249089 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:11.249105 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.250634 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.250642 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:11.250944 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:11.251070 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 22:10:11.251212 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 22:10:11.251337 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":422,"threads_started":1}
I20260927 22:10:11.251744 10853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e3fa6bc6ae84baa81411d2f45ebf840" candidate_uuid: "179737049c8d492b990cf2cc8817e494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf4342f2259649b69d3d545cf51f5969" is_pre_election: true
I20260927 22:10:11.252380 10853 raft_consensus.cc:2454] T 5e3fa6bc6ae84baa81411d2f45ebf840 P cf4342f2259649b69d3d545cf51f5969 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 179737049c8d492b990cf2cc8817e494 in term 0.
I20260927 22:10:11.252892 10987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e3fa6bc6ae84baa81411d2f45ebf840" candidate_uuid: "179737049c8d492b990cf2cc8817e494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4214deac0dbd4b5097029d85533ebba3" is_pre_election: true
I20260927 22:10:11.253254 11054 leader_election.cc:302] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 179737049c8d492b990cf2cc8817e494, cf4342f2259649b69d3d545cf51f5969; no voters:
I20260927 22:10:11.253328 10987 raft_consensus.cc:2454] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 4214deac0dbd4b5097029d85533ebba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 179737049c8d492b990cf2cc8817e494 in term 0.
I20260927 22:10:11.253620 11192 raft_consensus.cc:2781] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:10:11.253772 11192 raft_consensus.cc:482] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:10:11.253880 11192 raft_consensus.cc:3037] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:10:11.256146 11192 raft_consensus.cc:504] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4342f2259649b69d3d545cf51f5969" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41561 } } peers { permanent_uuid: "4214deac0dbd4b5097029d85533ebba3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43081 } } peers { permanent_uuid: "179737049c8d492b990cf2cc8817e494" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41231 } }
I20260927 22:10:11.256728 11192 leader_election.cc:288] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494 [CANDIDATE]: Term 1 election: Requested vote from peers cf4342f2259649b69d3d545cf51f5969 (127.0.97.1:41561), 4214deac0dbd4b5097029d85533ebba3 (127.0.97.2:43081)
I20260927 22:10:11.257256 10853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e3fa6bc6ae84baa81411d2f45ebf840" candidate_uuid: "179737049c8d492b990cf2cc8817e494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf4342f2259649b69d3d545cf51f5969"
I20260927 22:10:11.257248 10987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e3fa6bc6ae84baa81411d2f45ebf840" candidate_uuid: "179737049c8d492b990cf2cc8817e494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4214deac0dbd4b5097029d85533ebba3"
I20260927 22:10:11.257458 10853 raft_consensus.cc:3037] T 5e3fa6bc6ae84baa81411d2f45ebf840 P cf4342f2259649b69d3d545cf51f5969 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:10:11.257458 10987 raft_consensus.cc:3037] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 4214deac0dbd4b5097029d85533ebba3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:10:11.260684 10987 raft_consensus.cc:2454] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 4214deac0dbd4b5097029d85533ebba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 179737049c8d492b990cf2cc8817e494 in term 1.
I20260927 22:10:11.260684 10853 raft_consensus.cc:2454] T 5e3fa6bc6ae84baa81411d2f45ebf840 P cf4342f2259649b69d3d545cf51f5969 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 179737049c8d492b990cf2cc8817e494 in term 1.
I20260927 22:10:11.261158 11057 leader_election.cc:302] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 179737049c8d492b990cf2cc8817e494, 4214deac0dbd4b5097029d85533ebba3; no voters:
I20260927 22:10:11.261538 11192 raft_consensus.cc:2781] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:10:11.261986 11192 raft_consensus.cc:686] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494 [term 1 LEADER]: Becoming Leader. State: Replica: 179737049c8d492b990cf2cc8817e494, State: Running, Role: LEADER
I20260927 22:10:11.262537 11192 consensus_queue.cc:237] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4342f2259649b69d3d545cf51f5969" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41561 } } peers { permanent_uuid: "4214deac0dbd4b5097029d85533ebba3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43081 } } peers { permanent_uuid: "179737049c8d492b990cf2cc8817e494" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41231 } }
I20260927 22:10:11.267444 10709 catalog_manager.cc:5495] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494 reported cstate change: term changed from 0 to 1, leader changed from <none> to 179737049c8d492b990cf2cc8817e494 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "179737049c8d492b990cf2cc8817e494" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4342f2259649b69d3d545cf51f5969" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4214deac0dbd4b5097029d85533ebba3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "179737049c8d492b990cf2cc8817e494" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41231 } health_report { overall_health: HEALTHY } } }
I20260927 22:10:11.285945 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:11.280639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43362 (local address 127.0.97.3:41231)
0927 22:10:11.280811 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:11.280815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:11.280910 (+ 95us) server_negotiation.cc:407] Connection header received
0927 22:10:11.281079 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:11.281082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:11.281175 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:11.281303 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:10:11.281631 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.282426 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.283914 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.284342 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.284435 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:11.284949 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:10:11.285437 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:10:11.285546 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:11.285680 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 22:10:11.285753 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 22:10:11.290310 10899 tablet.cc:2369] T 5e3fa6bc6ae84baa81411d2f45ebf840 P cf4342f2259649b69d3d545cf51f5969: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:10:11.294214 10987 raft_consensus.cc:1261] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 4214deac0dbd4b5097029d85533ebba3 [term 1 FOLLOWER]: Refusing update from remote peer 179737049c8d492b990cf2cc8817e494: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:10:11.294217 10853 raft_consensus.cc:1261] T 5e3fa6bc6ae84baa81411d2f45ebf840 P cf4342f2259649b69d3d545cf51f5969 [term 1 FOLLOWER]: Refusing update from remote peer 179737049c8d492b990cf2cc8817e494: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:10:11.295308 11194 consensus_queue.cc:1035] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf4342f2259649b69d3d545cf51f5969" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:10:11.295704 11192 consensus_queue.cc:1035] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4214deac0dbd4b5097029d85533ebba3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:10:11.310398 11198 mvcc.cc:205] Tried to move back new op lower bound from 7334080558250864640 to 7334080558135975936. Current Snapshot: MvccSnapshot[applied={T|T < 7334080558250864640}]
I20260927 22:10:11.325767 10708 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:10:11.326099 10708 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:10:11.329826 10708 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 834bcc2e893a4d6888d7ad8c80e95069: Sending DeleteTablet for 3 replicas of tablet 5e3fa6bc6ae84baa81411d2f45ebf840
I20260927 22:10:11.330940 11101 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e3fa6bc6ae84baa81411d2f45ebf840 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:10:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43358
I20260927 22:10:11.330998 10833 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e3fa6bc6ae84baa81411d2f45ebf840 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:10:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37632
I20260927 22:10:11.331663 10967 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e3fa6bc6ae84baa81411d2f45ebf840 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:10:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43204
I20260927 22:10:11.331943 11208 tablet_replica.cc:330] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494: stopping tablet replica
I20260927 22:10:11.332528 11208 raft_consensus.cc:2229] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:10:11.333298 11208 raft_consensus.cc:2258] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:10:11.334683 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 10770
I20260927 22:10:11.335580 11208 ts_tablet_manager.cc:1905] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:10:11.342154 11210 tablet_replica.cc:330] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 4214deac0dbd4b5097029d85533ebba3: stopping tablet replica
I20260927 22:10:11.342653 11210 raft_consensus.cc:2229] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 4214deac0dbd4b5097029d85533ebba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:10:11.343231 11210 raft_consensus.cc:2258] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 4214deac0dbd4b5097029d85533ebba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:10:11.344465 11208 ts_tablet_manager.cc:1918] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:10:11.344695 11208 log.cc:1196] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/5e3fa6bc6ae84baa81411d2f45ebf840
I20260927 22:10:11.345369 11208 ts_tablet_manager.cc:1939] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 179737049c8d492b990cf2cc8817e494: Deleting consensus metadata
I20260927 22:10:11.347265 10695 catalog_manager.cc:4841] TS 179737049c8d492b990cf2cc8817e494 (127.0.97.3:41231): tablet 5e3fa6bc6ae84baa81411d2f45ebf840 (table test-table [id=b5a454c48b8f4c16873c31da286272a9]) successfully deleted
I20260927 22:10:11.347515 11210 ts_tablet_manager.cc:1905] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 4214deac0dbd4b5097029d85533ebba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:10:11.348598 11209 tablet_replica.cc:330] T 5e3fa6bc6ae84baa81411d2f45ebf840 P cf4342f2259649b69d3d545cf51f5969: stopping tablet replica
I20260927 22:10:11.349002 11209 raft_consensus.cc:2229] T 5e3fa6bc6ae84baa81411d2f45ebf840 P cf4342f2259649b69d3d545cf51f5969 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:10:11.349429 11209 raft_consensus.cc:2258] T 5e3fa6bc6ae84baa81411d2f45ebf840 P cf4342f2259649b69d3d545cf51f5969 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 22:10:11 dist-test-slave-mp03 krb5kdc[10664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:10:11.353370 11209 ts_tablet_manager.cc:1905] T 5e3fa6bc6ae84baa81411d2f45ebf840 P cf4342f2259649b69d3d545cf51f5969: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:10:11.354915 11210 ts_tablet_manager.cc:1918] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 4214deac0dbd4b5097029d85533ebba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:10:11.355072 11210 log.cc:1196] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 4214deac0dbd4b5097029d85533ebba3: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/5e3fa6bc6ae84baa81411d2f45ebf840
I20260927 22:10:11.355595 11210 ts_tablet_manager.cc:1939] T 5e3fa6bc6ae84baa81411d2f45ebf840 P 4214deac0dbd4b5097029d85533ebba3: Deleting consensus metadata
I20260927 22:10:11.357401 10696 catalog_manager.cc:4841] TS 4214deac0dbd4b5097029d85533ebba3 (127.0.97.2:43081): tablet 5e3fa6bc6ae84baa81411d2f45ebf840 (table test-table [id=b5a454c48b8f4c16873c31da286272a9]) successfully deleted
I20260927 22:10:11.360749 11209 ts_tablet_manager.cc:1918] T 5e3fa6bc6ae84baa81411d2f45ebf840 P cf4342f2259649b69d3d545cf51f5969: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:10:11.360958 11209 log.cc:1196] T 5e3fa6bc6ae84baa81411d2f45ebf840 P cf4342f2259649b69d3d545cf51f5969: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546912983837-388-0/minicluster-data/ts-0/wal/wals/5e3fa6bc6ae84baa81411d2f45ebf840
I20260927 22:10:11.361263 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:11.336029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37634 (local address 127.0.97.1:41561)
0927 22:10:11.336229 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:11.336238 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:11.340129 (+ 3891us) server_negotiation.cc:407] Connection header received
0927 22:10:11.340319 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:11.340324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:11.340431 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:11.340542 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:11.343562 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.344754 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.346691 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.347300 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.347412 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:11.353543 (+ 6131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:11.353555 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:11.353572 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:10:11.353622 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:11.357746 (+ 4124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:10:11.358700 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:10:11.358710 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:10:11.358714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:10:11.358894 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:10:11.359361 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:10:11.359366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:10:11.359369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:10:11.359830 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:10:11.360112 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:11.360455 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 22:10:11.360685 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:10:11.361562 11209 ts_tablet_manager.cc:1939] T 5e3fa6bc6ae84baa81411d2f45ebf840 P cf4342f2259649b69d3d545cf51f5969: Deleting consensus metadata
I20260927 22:10:11.361917 10893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:10:11.475148 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 10770
I20260927 22:10:11.475203 10693 catalog_manager.cc:4841] TS cf4342f2259649b69d3d545cf51f5969 (127.0.97.1:41561): tablet 5e3fa6bc6ae84baa81411d2f45ebf840 (table test-table [id=b5a454c48b8f4c16873c31da286272a9]) successfully deleted
I20260927 22:10:11.499315 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 10904
Sep 27 22:10:11 dist-test-slave-mp03 krb5kdc[10664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:10:11.516600 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:11.500115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43216 (local address 127.0.97.2:43081)
0927 22:10:11.500314 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:11.500321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:11.500343 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:10:11.500483 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:11.500489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:11.500574 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:11.500688 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:11.501983 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.503269 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.504455 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.505084 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.505212 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:11.509913 (+ 4701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:11.509941 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:11.509962 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:10:11.510022 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:11.513824 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:10:11.514463 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:10:11.514472 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:10:11.514477 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:10:11.514659 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:10:11.515091 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:10:11.515096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:10:11.515098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:10:11.515463 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:10:11.515745 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:11.516064 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 22:10:11.516275 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 22:10:11.517489 11027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:10:11.607393 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 10904
I20260927 22:10:11.630820 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 11038
Sep 27 22:10:11 dist-test-slave-mp03 krb5kdc[10664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:10:11.648098 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:11.631601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43372 (local address 127.0.97.3:41231)
0927 22:10:11.631769 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:11.631773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:11.631935 (+ 162us) server_negotiation.cc:407] Connection header received
0927 22:10:11.632146 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:11.632151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:11.632239 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:11.632348 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:11.633674 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.634507 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.635848 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:11.636417 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:11.636557 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:11.641122 (+ 4565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:11.641135 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:11.641151 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:10:11.641195 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:11.645158 (+ 3963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:10:11.645833 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:10:11.645845 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:10:11.645849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:10:11.646060 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:10:11.646422 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:10:11.646428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:10:11.646432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:10:11.646929 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:10:11.647260 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:11.647417 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 22:10:11.647648 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:10:11.648921 11161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:10:11.769546 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 11038
I20260927 22:10:11.790836 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 10677
I20260927 22:10:11.792043 10739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:10:11.899384 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 10677
2026-09-27T22:10:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4299 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:10:12 dist-test-slave-mp03 krb5kdc[11218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:10:12 dist-test-slave-mp03 krb5kdc[11218](info): set up 2 sockets
Sep 27 22:10:12 dist-test-slave-mp03 krb5kdc[11218](info): commencing operation
krb5kdc: starting...
W20260927 22:10:13.011495 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:10:13 dist-test-slave-mp03 krb5kdc[11218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:10:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:10:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260927 22:10:13.342453 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task3T1nfn/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/postgres -l logfile start
2026-09-27 22:10:15.938 UTC [11248] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 22:10:15.938 UTC [11248] LOG: listening on IPv4 address "127.0.97.20", port 33403
2026-09-27 22:10:15.942 UTC [11248] LOG: listening on Unix socket "/tmp/.s.PGSQL.33403"
2026-09-27 22:10:15.964 UTC [11251] LOG: database system was shut down at 2026-09-27 22:10:14 UTC
2026-09-27 22:10:15.969 UTC [11248] LOG: database system is ready to accept connections
I20260927 22:10:16.975071 388 mini_postgres.cc:96] Postgres bound to 33403
2026-09-27 22:10:16.992 UTC [11259] FATAL: database "slave" does not exist
127.0.97.20:33403 - accepting connections
I20260927 22:10:16.993335 388 mini_ranger.cc:161] Starting Ranger...
I20260927 22:10:17.020380 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 22:10:17.556358 388 mini_ranger.cc:87] Created ranger Postgres database
I20260927 22:10:17.556491 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ranger-admin
2026-09-27 22:10:17,861 [I] DB FLAVOR :POSTGRES
2026-09-27 22:10:17,862 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:10:17,862 [I] Checking connection..
2026-09-27 22:10:17,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 22:10:18,108 [I] Checking connection passed.
2026-09-27 22:10:18,108 [I] --------- Verifying version history table ---------
2026-09-27 22:10:18,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:10:18,370 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:10:18,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:10:18,619 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:10:18,619 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 22:10:18,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33403/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 22:10:18.858 UTC [11323] WARNING: there is no transaction in progress
2026-09-27 22:10:18,865 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 22:10:18,865 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:10:19,125 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 22:10:19,125 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:10:19,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 22:10:19,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 22:10:19,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mp03.c.gcp-upstream.internal','N') ;"
2026-09-27 22:10:19,863 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:10:20,126 [I] Table x_portal_user does not exist in database ranger
2026-09-27 22:10:20,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:10:20,366 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 22:10:20,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:10:20,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:10:20,856 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 22:10:20,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33403/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 22:10:21.870 UTC [11431] WARNING: there is no transaction in progress
2026-09-27 22:10:21.889 UTC [11431] WARNING: there is no transaction in progress
2026-09-27 22:10:21.907 UTC [11431] WARNING: there is no transaction in progress
2026-09-27 22:10:21.925 UTC [11431] WARNING: there is no transaction in progress
2026-09-27 22:10:21.943 UTC [11431] WARNING: there is no transaction in progress
2026-09-27 22:10:21.962 UTC [11431] WARNING: there is no transaction in progress
2026-09-27 22:10:22.024 UTC [11431] WARNING: there is no transaction in progress
2026-09-27 22:10:22.032 UTC [11431] WARNING: there is no transaction in progress
2026-09-27 22:10:22.046 UTC [11431] WARNING: there is no transaction in progress
2026-09-27 22:10:22.057 UTC [11431] WARNING: there is no transaction in progress
2026-09-27 22:10:22.068 UTC [11431] WARNING: there is no transaction in progress
2026-09-27 22:10:22.082 UTC [11431] WARNING: there is no transaction in progress
2026-09-27 22:10:22.092 UTC [11431] WARNING: there is no transaction in progress
2026-09-27 22:10:22.099 UTC [11431] WARNING: there is no transaction in progress
2026-09-27 22:10:22.106 UTC [11431] WARNING: there is no transaction in progress
2026-09-27 22:10:22.113 UTC [11431] WARNING: there is no transaction in progress
2026-09-27 22:10:22,789 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 22:10:22,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 22:10:23,038 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 22:10:23,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:10:23,308 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 22:10:23,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:10:23,544 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 22:10:23,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:10:23,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:10:24,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mp03.c.gcp-upstream.internal';"
2026-09-27 22:10:24,313 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 22:10:24,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 22:10:24,564 [I] DB_PATCHES have already been applied
I20260927 22:10:24.570102 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ranger-admin:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task3T1nfn/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 22:10:24.570191 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260927 22:10:24.585856 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:36419
Sep 27, 2026 10:10:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:10:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:10:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 10:10:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 10:10:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 10:10:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 10:10:25 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 22:10:25.737872 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.745s user 0.002s sys 0.039s
Sep 27, 2026 10:10:46 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 10:10:46 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 10:10:46 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 10:10:46 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 10:10:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 10:10:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 10:10:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 10:10:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 10:10:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 10:10:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 10:10:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 10:10:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 10:10:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 10:10:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 10:10:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 10:10:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 10:10:46 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 10:10:47 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 22:10:48.090667 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 22:10:48 dist-test-slave-mp03 krb5kdc[11218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547048, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 22:10:48.277966 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 22:10:48.299953 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 22:10:49.200728 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 22:10:49.200905 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ranger-kms
I20260927 22:10:49.201026 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:33403
2026-09-27 22:10:49,647 [I] DB FLAVOR :POSTGRES
2026-09-27 22:10:49,648 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:10:49,648 [I] Checking connection
2026-09-27 22:10:49,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33403/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 22:10:49,879 [I] connection success
2026-09-27 22:10:49,879 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 22:10:49,879 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 22:10:49,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33403/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 22:10:50,110 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 22:10:50,110 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:10:50,110 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 22:10:50,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33403/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 22:10:50,384 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 22:10:50.884338 388 mini_ranger_kms.cc:310] Created kms service
I20260927 22:10:51.077126 388 mini_ranger_kms.cc:326] Created kudu user
I20260927 22:10:51.127287 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 22:10:51 dist-test-slave-mp03 krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547048, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:10:51.566052 388 mini_ranger_kms.cc:384] Added ranger policy
I20260927 22:10:51.566352 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ranger-kms:/tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 22:10:51.566406 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260927 22:10:51.585556 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11612
I20260927 22:10:51.585763 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:50855
Sep 27, 2026 10:10:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:10:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:10:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 10:10:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 10:10:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 10:10:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 22:10:53.869400 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.591s user 0.122s sys 0.343s
I20260927 22:10:53.869673 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 22:10:53.869907 388 mini_ranger_kms.cc:401] 127.0.97.20:50855/kms/v1/keys
Sep 27 22:10:53 dist-test-slave-mp03 krb5kdc[11218](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790547053, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 22:10:54 dist-test-slave-mp03 krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547053, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 22:10:54 dist-test-slave-mp03 krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547053, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 22:10:56 dist-test-slave-mp03 krb5kdc[11218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:10:57.035430 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42551
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42551
--ranger_config_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50855/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:10:57.390296 11662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:10:57.390669 11662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:10:57.390832 11662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:10:57.398232 11662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:10:57.398386 11662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:10:57.398511 11662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:10:57.398628 11662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:10:57.398721 11662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:10:57.409538 11662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:50855/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42551
--ranger_config_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:10:57.411010 11662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:10:57.412428 11662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:10:57.425088 11667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:57.425129 11668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:57.425156 11670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:10:57.425947 11662 server_base.cc:1034] running on GCE node
I20260927 22:10:57.426851 11662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:10:57.428356 11662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:10:57.429555 11662 hybrid_clock.cc:648] HybridClock initialized: now 1790547057429516 us; error 35 us; skew 500 ppm
I20260927 22:10:57.429929 11662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:10:57 dist-test-slave-mp03 krb5kdc[11218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:10:57.433650 11662 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:10:57.435385 11662 webserver.cc:460] Webserver started at http://127.0.97.62:34013/ using document root <none> and password file <none>
I20260927 22:10:57.436086 11662 fs_manager.cc:359] Metadata directory not provided
I20260927 22:10:57.436194 11662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:10:57.436482 11662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:10:57 dist-test-slave-mp03 krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:10:57.488256 11662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "c33ccda1390b454ea286af5d8182d3a2"
format_stamp: "Formatted at 2026-09-27 22:10:57 on dist-test-slave-mp03"
server_key: "24cdb768c922b91cc459528039aae60e"
server_key_iv: "34f03ed09a626776688ef61fad12a232"
server_key_version: "kuduclusterkey@0"
I20260927 22:10:57.489296 11662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "c33ccda1390b454ea286af5d8182d3a2"
format_stamp: "Formatted at 2026-09-27 22:10:57 on dist-test-slave-mp03"
server_key: "24cdb768c922b91cc459528039aae60e"
server_key_iv: "34f03ed09a626776688ef61fad12a232"
server_key_version: "kuduclusterkey@0"
I20260927 22:10:57.496527 11662 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260927 22:10:57.517388 11677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:57.518986 11662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:10:57.519170 11662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "c33ccda1390b454ea286af5d8182d3a2"
format_stamp: "Formatted at 2026-09-27 22:10:57 on dist-test-slave-mp03"
server_key: "24cdb768c922b91cc459528039aae60e"
server_key_iv: "34f03ed09a626776688ef61fad12a232"
server_key_version: "kuduclusterkey@0"
I20260927 22:10:57.519397 11662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:10:57.554184 11662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:10:57.558745 11662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:10:57.559051 11662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:10:57.577980 11662 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42551
I20260927 22:10:57.578002 11728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42551 every 8 connection(s)
I20260927 22:10:57.579440 11662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:10:57.584054 11729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:10:57.586680 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 11662
I20260927 22:10:57.586920 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:10:57.587810 388 external_mini_cluster.cc:1428] Setting key 0ee79d42e3089336ee7378aa1380cc24
I20260927 22:10:57.598762 11729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c33ccda1390b454ea286af5d8182d3a2: Bootstrap starting.
I20260927 22:10:57.602730 11729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c33ccda1390b454ea286af5d8182d3a2: Neither blocks nor log segments found. Creating new log.
I20260927 22:10:57.603833 11729 log.cc:824] T 00000000000000000000000000000000 P c33ccda1390b454ea286af5d8182d3a2: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:10:57 dist-test-slave-mp03 krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:10:57.607257 11729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c33ccda1390b454ea286af5d8182d3a2: No bootstrap required, opened a new log
I20260927 22:10:57.613493 11732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:57.590050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40350 (local address 127.0.97.62:42551)
0927 22:10:57.590922 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:57.590939 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:57.591002 (+ 63us) server_negotiation.cc:407] Connection header received
0927 22:10:57.593590 (+ 2588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:57.593627 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:57.594222 (+ 595us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:57.594745 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:57.595928 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:57.598468 (+ 2540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:57.599857 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:57.601064 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:57.601254 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:57.605676 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:57.605684 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:57.605699 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:10:57.605731 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:57.608797 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:10:57.609691 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:10:57.609697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:10:57.609704 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:10:57.609853 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:10:57.610279 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:10:57.610282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:10:57.610284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:10:57.610891 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:10:57.611132 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:57.611783 (+ 651us) server_negotiation.cc:299] Negotiation successful
0927 22:10:57.612164 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":381,"threads_started":1}
I20260927 22:10:57.617434 11729 raft_consensus.cc:348] T 00000000000000000000000000000000 P c33ccda1390b454ea286af5d8182d3a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33ccda1390b454ea286af5d8182d3a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42551 } }
I20260927 22:10:57.617848 11729 raft_consensus.cc:374] T 00000000000000000000000000000000 P c33ccda1390b454ea286af5d8182d3a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:10:57.617956 11729 raft_consensus.cc:729] T 00000000000000000000000000000000 P c33ccda1390b454ea286af5d8182d3a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33ccda1390b454ea286af5d8182d3a2, State: Initialized, Role: FOLLOWER
I20260927 22:10:57.618739 11729 consensus_queue.cc:260] T 00000000000000000000000000000000 P c33ccda1390b454ea286af5d8182d3a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33ccda1390b454ea286af5d8182d3a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42551 } }
I20260927 22:10:57.618996 11729 raft_consensus.cc:388] T 00000000000000000000000000000000 P c33ccda1390b454ea286af5d8182d3a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:10:57.619143 11729 raft_consensus.cc:482] T 00000000000000000000000000000000 P c33ccda1390b454ea286af5d8182d3a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:10:57.619347 11729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c33ccda1390b454ea286af5d8182d3a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:10:57.622135 11729 raft_consensus.cc:504] T 00000000000000000000000000000000 P c33ccda1390b454ea286af5d8182d3a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33ccda1390b454ea286af5d8182d3a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42551 } }
I20260927 22:10:57.622807 11729 leader_election.cc:302] T 00000000000000000000000000000000 P c33ccda1390b454ea286af5d8182d3a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c33ccda1390b454ea286af5d8182d3a2; no voters:
I20260927 22:10:57.623495 11729 leader_election.cc:288] T 00000000000000000000000000000000 P c33ccda1390b454ea286af5d8182d3a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:10:57.623622 11734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c33ccda1390b454ea286af5d8182d3a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:10:57.624660 11734 raft_consensus.cc:686] T 00000000000000000000000000000000 P c33ccda1390b454ea286af5d8182d3a2 [term 1 LEADER]: Becoming Leader. State: Replica: c33ccda1390b454ea286af5d8182d3a2, State: Running, Role: LEADER
I20260927 22:10:57.625211 11734 consensus_queue.cc:237] T 00000000000000000000000000000000 P c33ccda1390b454ea286af5d8182d3a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33ccda1390b454ea286af5d8182d3a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42551 } }
I20260927 22:10:57.625946 11729 sys_catalog.cc:564] T 00000000000000000000000000000000 P c33ccda1390b454ea286af5d8182d3a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:10:57.629135 11736 sys_catalog.cc:455] T 00000000000000000000000000000000 P c33ccda1390b454ea286af5d8182d3a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c33ccda1390b454ea286af5d8182d3a2. Latest consensus state: current_term: 1 leader_uuid: "c33ccda1390b454ea286af5d8182d3a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33ccda1390b454ea286af5d8182d3a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42551 } } }
I20260927 22:10:57.629093 11735 sys_catalog.cc:455] T 00000000000000000000000000000000 P c33ccda1390b454ea286af5d8182d3a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c33ccda1390b454ea286af5d8182d3a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33ccda1390b454ea286af5d8182d3a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42551 } } }
I20260927 22:10:57.629534 11736 sys_catalog.cc:458] T 00000000000000000000000000000000 P c33ccda1390b454ea286af5d8182d3a2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:10:57.629534 11735 sys_catalog.cc:458] T 00000000000000000000000000000000 P c33ccda1390b454ea286af5d8182d3a2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:10:57.631145 11742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:10:57.637108 11742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:10:57.637840 11729 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 22:10:57.657812 11742 catalog_manager.cc:1347] Generated new cluster ID: cda7afc2cbcf43e099a733bf7ed05f0a
I20260927 22:10:57.657985 11742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:10:57.672478 11742 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:10:57.675037 11742 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:10:57.686640 11742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c33ccda1390b454ea286af5d8182d3a2: Generated new TSK 0
I20260927 22:10:57.687819 11742 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:10:58.442365 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50855/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42551
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:10:58 dist-test-slave-mp03 krb5kdc[11218](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790547058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 22:10:58.862169 11774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:10:58.862571 11774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:10:58.862756 11774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:10:58.870540 11774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:10:58.870739 11774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:10:58.870950 11774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:10:58.881330 11774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:50855/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:10:58.883460 11774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:10:58.885236 11774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:10:58.904026 11785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:58.905603 11787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:58.906813 11784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:10:58.907255 11774 server_base.cc:1034] running on GCE node
I20260927 22:10:58.908078 11774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:10:58.909287 11774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:10:58.910712 11774 hybrid_clock.cc:648] HybridClock initialized: now 1790547058910527 us; error 79 us; skew 500 ppm
I20260927 22:10:58.911201 11774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:10:58 dist-test-slave-mp03 krb5kdc[11218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:10:58.915843 11774 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:10:58.917860 11774 webserver.cc:460] Webserver started at http://127.0.97.1:41977/ using document root <none> and password file <none>
I20260927 22:10:58.918736 11774 fs_manager.cc:359] Metadata directory not provided
I20260927 22:10:58.918897 11774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:10:58.919262 11774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:10:58 dist-test-slave-mp03 krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:10:58.939966 11774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "e34aa6a29e5e40b9ad217e8e0d0c1e22"
format_stamp: "Formatted at 2026-09-27 22:10:58 on dist-test-slave-mp03"
server_key: "53e72147b3b0aadc28b632d83433e8f9"
server_key_iv: "489d30858e2be8a77bc1e1655e298375"
server_key_version: "kuduclusterkey@0"
I20260927 22:10:58.941756 11774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e34aa6a29e5e40b9ad217e8e0d0c1e22"
format_stamp: "Formatted at 2026-09-27 22:10:58 on dist-test-slave-mp03"
server_key: "53e72147b3b0aadc28b632d83433e8f9"
server_key_iv: "489d30858e2be8a77bc1e1655e298375"
server_key_version: "kuduclusterkey@0"
I20260927 22:10:58.965036 11774 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.004s sys 0.020s
I20260927 22:10:59.003319 11794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:59.005571 11774 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.008s sys 0.008s
I20260927 22:10:59.005864 11774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "e34aa6a29e5e40b9ad217e8e0d0c1e22"
format_stamp: "Formatted at 2026-09-27 22:10:58 on dist-test-slave-mp03"
server_key: "53e72147b3b0aadc28b632d83433e8f9"
server_key_iv: "489d30858e2be8a77bc1e1655e298375"
server_key_version: "kuduclusterkey@0"
I20260927 22:10:59.006237 11774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:10:59.030093 11774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:10:59.035904 11774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:10:59.036671 11774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:10:59.037995 11774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:10:59.039893 11774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:10:59.039994 11774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:59.040129 11774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:10:59.040203 11774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:59.115129 11906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38353 every 8 connection(s)
I20260927 22:10:59.115321 11774 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38353
I20260927 22:10:59.116693 11774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:10:59.119169 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 11774
I20260927 22:10:59.119410 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:10:59.120278 388 external_mini_cluster.cc:1428] Setting key 79cd0b6d999a80f6029c18f21e19c2d3
Sep 27 22:10:59 dist-test-slave-mp03 krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 22:10:59.185570 11910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:59.122078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46237 (local address 127.0.97.62:42551)
0927 22:10:59.152490 (+ 30412us) server_negotiation.cc:206] Beginning negotiation
0927 22:10:59.152498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:10:59.153762 (+ 1264us) server_negotiation.cc:407] Connection header received
0927 22:10:59.163729 (+ 9967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:10:59.163738 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:10:59.163860 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:10:59.163954 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:10:59.167289 (+ 3335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:59.168677 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:59.171380 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:10:59.172090 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:10:59.172195 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:59.177168 (+ 4973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:10:59.177176 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:10:59.177182 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:10:59.177231 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:10:59.181252 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:10:59.182349 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:10:59.182355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:10:59.182360 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:10:59.182536 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:10:59.183207 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:10:59.183213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:10:59.183217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:10:59.183581 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:10:59.183716 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 22:10:59.184768 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0927 22:10:59.185014 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30258,"thread_start_us":30442,"threads_started":1}
I20260927 22:10:59.187960 11909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:10:59.121478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42551 (local address 127.0.97.1:46237)
0927 22:10:59.153314 (+ 31836us) negotiation.cc:107] Waiting for socket to connect
0927 22:10:59.153389 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 22:10:59.156225 (+ 2836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:10:59.164608 (+ 8383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:10:59.164622 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:10:59.166010 (+ 1388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:10:59.167029 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:59.167073 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:59.168919 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:59.168925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:59.171197 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:10:59.171212 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:10:59.172572 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:10:59.172580 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:10:59.172888 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:10:59.173745 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:10:59.173777 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:10:59.176823 (+ 3046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:10:59.181476 (+ 4653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:10:59.181487 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:10:59.181510 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:10:59.182185 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:10:59.182739 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:10:59.182744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:10:59.182748 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:10:59.183042 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:10:59.183815 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:10:59.183824 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:10:59.184309 (+ 485us) client_negotiation.cc:769] Sending connection context
0927 22:10:59.185885 (+ 1576us) client_negotiation.cc:240] Negotiation successful
0927 22:10:59.186481 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31440,"thread_start_us":41029,"threads_started":1}
I20260927 22:10:59.190083 11907 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42551
I20260927 22:10:59.190438 11907 heartbeater.cc:464] Registering TS with master...
I20260927 22:10:59.191414 11907 heartbeater.cc:511] Master 127.0.97.62:42551 requested a full tablet report, sending...
I20260927 22:10:59.193934 11694 ts_manager.cc:194] Registered new tserver with Master: e34aa6a29e5e40b9ad217e8e0d0c1e22 (127.0.97.1:38353)
I20260927 22:10:59.196084 11694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
W20260927 22:10:59.225245 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.2: real 0.102s user 0.002s sys 0.015s
I20260927 22:10:59.226183 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50855/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42551
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:10:59 dist-test-slave-mp03 krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 22:10:59 dist-test-slave-mp03 krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260927 22:10:59.746049 11913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:10:59.746562 11913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:10:59.746770 11913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:10:59.758137 11913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:10:59.758348 11913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:10:59.758562 11913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:10:59.774478 11913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:50855/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:10:59.776528 11913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:10:59.778405 11913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:10:59.814287 11920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:10:59.815821 11921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:10:59.825052 11913 server_base.cc:1034] running on GCE node
W20260927 22:10:59.831182 11923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:10:59.832329 11913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:10:59.848587 11913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:10:59.857015 11913 hybrid_clock.cc:648] HybridClock initialized: now 1790547059854734 us; error 2217 us; skew 500 ppm
I20260927 22:10:59.857669 11913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:10:59 dist-test-slave-mp03 krb5kdc[11218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:10:59.862525 11913 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:10:59.872421 11913 webserver.cc:460] Webserver started at http://127.0.97.2:45099/ using document root <none> and password file <none>
I20260927 22:10:59.873283 11913 fs_manager.cc:359] Metadata directory not provided
I20260927 22:10:59.873450 11913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:10:59.873811 11913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:10:59 dist-test-slave-mp03 krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:10:59.896100 11913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "62e9262d25ee40f39310c1b09323a676"
format_stamp: "Formatted at 2026-09-27 22:10:59 on dist-test-slave-mp03"
server_key: "a8237761f5c15b9b38ea4b1368f29e11"
server_key_iv: "2aca9a3647655a945918ee0766a5cc28"
server_key_version: "kuduclusterkey@0"
I20260927 22:10:59.897224 11913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "62e9262d25ee40f39310c1b09323a676"
format_stamp: "Formatted at 2026-09-27 22:10:59 on dist-test-slave-mp03"
server_key: "a8237761f5c15b9b38ea4b1368f29e11"
server_key_iv: "2aca9a3647655a945918ee0766a5cc28"
server_key_version: "kuduclusterkey@0"
I20260927 22:10:59.903983 11913 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260927 22:10:59.920565 11937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:59.922282 11913 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 22:10:59.922585 11913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "62e9262d25ee40f39310c1b09323a676"
format_stamp: "Formatted at 2026-09-27 22:10:59 on dist-test-slave-mp03"
server_key: "a8237761f5c15b9b38ea4b1368f29e11"
server_key_iv: "2aca9a3647655a945918ee0766a5cc28"
server_key_version: "kuduclusterkey@0"
I20260927 22:10:59.922858 11913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:10:59.949193 11913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:10:59.953622 11913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:10:59.953941 11913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:10:59.955035 11913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:10:59.956501 11913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:10:59.956605 11913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:10:59.956758 11913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:10:59.957226 11913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:11:00.000485 11913 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32899
I20260927 22:11:00.000504 12049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32899 every 8 connection(s)
I20260927 22:11:00.002267 11913 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:11:00.004175 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 11913
I20260927 22:11:00.004383 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:11:00.005105 388 external_mini_cluster.cc:1428] Setting key 82095d4bdfeb71b112c0613942d8b43b
Sep 27 22:11:00 dist-test-slave-mp03 krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 22:11:00.055190 12052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:00.005997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42551 (local address 127.0.97.2:55131)
0927 22:11:00.027572 (+ 21575us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:00.027641 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:00.035003 (+ 7362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:00.036228 (+ 1225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:00.036241 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:00.036878 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:11:00.037688 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:00.037713 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.039203 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.039209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:00.041300 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:00.041325 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.042563 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.042570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:00.042873 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:00.043683 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:11:00.043714 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:11:00.046489 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:11:00.050412 (+ 3923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:00.050422 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:00.050442 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:00.051051 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:00.051493 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:00.051498 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:00.051501 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:00.051778 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:00.052407 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:11:00.052415 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:11:00.052833 (+ 418us) client_negotiation.cc:769] Sending connection context
0927 22:11:00.053127 (+ 294us) client_negotiation.cc:240] Negotiation successful
0927 22:11:00.053660 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21128,"thread_start_us":26922,"threads_started":1}
I20260927 22:11:00.056335 12053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:00.006915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55131 (local address 127.0.97.62:42551)
0927 22:11:00.032972 (+ 26057us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:00.032979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:00.033010 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:11:00.035511 (+ 2501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:00.035517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:00.035620 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:00.035700 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:00.037894 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.039010 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.041498 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.042156 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.042281 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:00.046779 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:00.046787 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:00.046792 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:00.046833 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:00.050222 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:00.051193 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:00.051199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:00.051202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:00.051370 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:00.051920 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:00.051924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:00.051927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:00.052262 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:00.052399 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:00.055731 (+ 3332us) server_negotiation.cc:299] Negotiation successful
0927 22:11:00.055974 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25910,"thread_start_us":27182,"threads_started":1}
I20260927 22:11:00.058452 12050 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42551
I20260927 22:11:00.058851 12050 heartbeater.cc:464] Registering TS with master...
I20260927 22:11:00.059800 12050 heartbeater.cc:511] Master 127.0.97.62:42551 requested a full tablet report, sending...
I20260927 22:11:00.061439 11694 ts_manager.cc:194] Registered new tserver with Master: 62e9262d25ee40f39310c1b09323a676 (127.0.97.2:32899)
I20260927 22:11:00.062464 11694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:11:00.092581 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50855/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42551
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:11:00.200083 11907 heartbeater.cc:503] Master 127.0.97.62:42551 was elected leader, sending a full tablet report...
W20260927 22:11:00.445819 12056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:11:00.446332 12056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:11:00.446717 12056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:11:00.455331 12056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:11:00.455499 12056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:11:00.455776 12056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:11:00.467756 12056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:50855/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:11:00.469260 12056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:11:00.470846 12056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:11:00.485533 12061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:11:00.485590 12064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:11:00.485503 12062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:11:00.486382 12056 server_base.cc:1034] running on GCE node
I20260927 22:11:00.487054 12056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:11:00.488027 12056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:11:00.489291 12056 hybrid_clock.cc:648] HybridClock initialized: now 1790547060489231 us; error 45 us; skew 500 ppm
I20260927 22:11:00.489625 12056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:11:00 dist-test-slave-mp03 krb5kdc[11218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:11:00.492828 12056 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:11:00.494329 12056 webserver.cc:460] Webserver started at http://127.0.97.3:39301/ using document root <none> and password file <none>
I20260927 22:11:00.495074 12056 fs_manager.cc:359] Metadata directory not provided
I20260927 22:11:00.495211 12056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:11:00.495519 12056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:11:00 dist-test-slave-mp03 krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:11:00.513240 12056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "c6ebb84b014c42e18cfcae21a28d61c7"
format_stamp: "Formatted at 2026-09-27 22:11:00 on dist-test-slave-mp03"
server_key: "bf097fb7c6c1ef75ae6a812717f9b01b"
server_key_iv: "1b8fe94050f000a4706b57891c04148e"
server_key_version: "kuduclusterkey@0"
I20260927 22:11:00.514340 12056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c6ebb84b014c42e18cfcae21a28d61c7"
format_stamp: "Formatted at 2026-09-27 22:11:00 on dist-test-slave-mp03"
server_key: "bf097fb7c6c1ef75ae6a812717f9b01b"
server_key_iv: "1b8fe94050f000a4706b57891c04148e"
server_key_version: "kuduclusterkey@0"
I20260927 22:11:00.521004 12056 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 22:11:00.543568 12071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:11:00.545518 12056 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.000s sys 0.004s
I20260927 22:11:00.545698 12056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "c6ebb84b014c42e18cfcae21a28d61c7"
format_stamp: "Formatted at 2026-09-27 22:11:00 on dist-test-slave-mp03"
server_key: "bf097fb7c6c1ef75ae6a812717f9b01b"
server_key_iv: "1b8fe94050f000a4706b57891c04148e"
server_key_version: "kuduclusterkey@0"
I20260927 22:11:00.545946 12056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:11:00.571398 12056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:11:00.575558 12056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:11:00.575825 12056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:11:00.576725 12056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:11:00.578171 12056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:11:00.578249 12056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:11:00.578379 12056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:11:00.578433 12056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:11:00.616046 12056 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37085
I20260927 22:11:00.616080 12183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37085 every 8 connection(s)
I20260927 22:11:00.617448 12056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:11:00.622041 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 12056
I20260927 22:11:00.622224 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:11:00.622745 388 external_mini_cluster.cc:1428] Setting key 9523559decebc55f8440ab0d3dd39a31
Sep 27 22:11:00 dist-test-slave-mp03 krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:11:00.639225 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:00.620252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60377 (local address 127.0.97.62:42551)
0927 22:11:00.620799 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:00.620811 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:00.621586 (+ 775us) server_negotiation.cc:407] Connection header received
0927 22:11:00.624445 (+ 2859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:00.624450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:00.624545 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:00.624663 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:00.626299 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.627208 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.628850 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.629265 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.629366 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:00.632989 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:00.632996 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:00.633000 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:00.633033 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:00.635815 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:00.636600 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:00.636605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:00.636608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:00.636808 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:00.637320 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:00.637326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:00.637329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:00.637760 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:00.637925 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:00.638638 (+ 713us) server_negotiation.cc:299] Negotiation successful
0927 22:11:00.638872 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":297,"threads_started":1}
I20260927 22:11:00.640306 12187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:00.620673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42551 (local address 127.0.97.3:60377)
0927 22:11:00.621389 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:00.621432 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:00.624156 (+ 2724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:00.624857 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:00.624865 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:00.625395 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:11:00.626110 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:00.626133 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.627344 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.627348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:00.628728 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:00.628738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.629463 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.629467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:00.629657 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:00.630347 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:11:00.630373 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:11:00.632749 (+ 2376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:11:00.635955 (+ 3206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:00.635962 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:00.635976 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:00.636429 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:00.636946 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:00.636952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:00.636955 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:00.637196 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:00.637897 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:11:00.637903 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:11:00.638252 (+ 349us) client_negotiation.cc:769] Sending connection context
0927 22:11:00.638537 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 22:11:00.639092 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":295,"threads_started":1}
I20260927 22:11:00.641697 12184 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42551
I20260927 22:11:00.642001 12184 heartbeater.cc:464] Registering TS with master...
I20260927 22:11:00.642755 12184 heartbeater.cc:511] Master 127.0.97.62:42551 requested a full tablet report, sending...
I20260927 22:11:00.644124 11694 ts_manager.cc:194] Registered new tserver with Master: c6ebb84b014c42e18cfcae21a28d61c7 (127.0.97.3:37085)
I20260927 22:11:00.644960 11694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60377
I20260927 22:11:00.651504 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:11:00.670961 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:00.656957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34520 (local address 127.0.97.62:42551)
0927 22:11:00.657164 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:00.657169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:00.657551 (+ 382us) server_negotiation.cc:407] Connection header received
0927 22:11:00.657711 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:00.657715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:00.657807 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:00.657916 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:00.659458 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.660360 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.661732 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.662244 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.662374 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:00.664903 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:00.664907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:00.664910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:00.664947 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:00.667639 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:00.668301 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:00.668305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:00.668307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:00.668499 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:00.669002 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:00.669007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:00.669011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:00.669477 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:00.669760 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:00.670382 (+ 622us) server_negotiation.cc:299] Negotiation successful
0927 22:11:00.670634 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:11:00.675027 11694 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:11:00.682219 11694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:11:00.711575 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:00.694396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38353 (local address 127.0.0.1:46440)
0927 22:11:00.695351 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:00.695392 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:00.695575 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:00.696613 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:00.696621 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:00.696652 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:11:00.699070 (+ 2418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:00.699087 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.702946 (+ 3859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.702956 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:00.707619 (+ 4663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:00.707651 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.710511 (+ 2860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.710520 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:00.710760 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:00.710817 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 22:11:00.710940 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 22:11:00.711145 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":546,"threads_started":1}
I20260927 22:11:00.712054 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:00.694629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46440 (local address 127.0.97.1:38353)
0927 22:11:00.695548 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:00.695558 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:00.695587 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:11:00.695815 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:00.695826 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:00.696120 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:00.696322 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:11:00.699334 (+ 3012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.702746 (+ 3412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.707893 (+ 5147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.710008 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.710179 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:00.711407 (+ 1228us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:00.711496 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 22:11:00.711687 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":573,"threads_started":1}
I20260927 22:11:00.715758 11842 tablet_service.cc:1467] Processing CreateTablet for tablet 85a53876196647fb99f14159e5dbbdd7 (DEFAULT_TABLE table=test-table [id=189198890d23485c9b013b9e5823cec5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:11:00.717307 11842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85a53876196647fb99f14159e5dbbdd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:11:00.718679 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:00.695755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32899 (local address 127.0.0.1:37644)
0927 22:11:00.696857 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:00.696885 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:00.697057 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:00.711106 (+ 14049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:00.711113 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:00.711140 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:11:00.711513 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:00.711522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.713639 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.713647 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:00.716038 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:00.716052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.717860 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.717868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:00.718042 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:00.718074 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:11:00.718178 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 22:11:00.718314 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":277,"threads_started":1}
I20260927 22:11:00.718799 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:00.696017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37644 (local address 127.0.97.2:32899)
0927 22:11:00.710456 (+ 14439us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:00.710464 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:00.710488 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:11:00.710582 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:00.710609 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:00.710881 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:00.711889 (+ 1008us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:11:00.712138 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.713512 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.716216 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.717657 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.717794 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:00.717975 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:00.718222 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 22:11:00.718395 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14258,"thread_start_us":14140,"threads_started":1}
I20260927 22:11:00.719678 12202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:00.696862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37085 (local address 127.0.0.1:51800)
0927 22:11:00.709615 (+ 12753us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:00.709633 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:00.709773 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:00.710766 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:00.710771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:00.710795 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:11:00.711275 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:00.711305 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.714349 (+ 3044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.714354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:00.716875 (+ 2521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:00.716891 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.718922 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.718930 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:00.719092 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:00.719135 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 22:11:00.719212 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 22:11:00.719331 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12634,"mutex_wait_us":1,"thread_start_us":12516,"threads_started":1}
I20260927 22:11:00.719743 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:00.696630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51800 (local address 127.0.97.3:37085)
0927 22:11:00.709177 (+ 12547us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:00.709190 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:00.709743 (+ 553us) server_negotiation.cc:407] Connection header received
0927 22:11:00.709949 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:00.709958 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:00.710259 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:00.710474 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:11:00.711525 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.714215 (+ 2690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.717048 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.718420 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.718540 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:00.718759 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:00.719265 (+ 506us) server_negotiation.cc:299] Negotiation successful
0927 22:11:00.719429 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12288,"thread_start_us":13243,"threads_started":1}
I20260927 22:11:00.722368 11985 tablet_service.cc:1467] Processing CreateTablet for tablet 85a53876196647fb99f14159e5dbbdd7 (DEFAULT_TABLE table=test-table [id=189198890d23485c9b013b9e5823cec5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:11:00.723942 11985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85a53876196647fb99f14159e5dbbdd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:11:00.723779 12119 tablet_service.cc:1467] Processing CreateTablet for tablet 85a53876196647fb99f14159e5dbbdd7 (DEFAULT_TABLE table=test-table [id=189198890d23485c9b013b9e5823cec5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:11:00.725021 12119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85a53876196647fb99f14159e5dbbdd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:11:00.732234 12203 tablet_bootstrap.cc:492] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22: Bootstrap starting.
I20260927 22:11:00.736829 12204 tablet_bootstrap.cc:492] T 85a53876196647fb99f14159e5dbbdd7 P 62e9262d25ee40f39310c1b09323a676: Bootstrap starting.
I20260927 22:11:00.736884 12203 tablet_bootstrap.cc:654] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22: Neither blocks nor log segments found. Creating new log.
I20260927 22:11:00.738076 12203 log.cc:824] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22: Log is configured to *not* fsync() on all Append() calls
I20260927 22:11:00.738852 12205 tablet_bootstrap.cc:492] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7: Bootstrap starting.
I20260927 22:11:00.740582 12204 tablet_bootstrap.cc:654] T 85a53876196647fb99f14159e5dbbdd7 P 62e9262d25ee40f39310c1b09323a676: Neither blocks nor log segments found. Creating new log.
I20260927 22:11:00.741986 12204 log.cc:824] T 85a53876196647fb99f14159e5dbbdd7 P 62e9262d25ee40f39310c1b09323a676: Log is configured to *not* fsync() on all Append() calls
I20260927 22:11:00.742041 12203 tablet_bootstrap.cc:492] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22: No bootstrap required, opened a new log
I20260927 22:11:00.742295 12203 ts_tablet_manager.cc:1397] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260927 22:11:00.742501 12205 tablet_bootstrap.cc:654] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7: Neither blocks nor log segments found. Creating new log.
I20260927 22:11:00.743721 12205 log.cc:824] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7: Log is configured to *not* fsync() on all Append() calls
I20260927 22:11:00.744870 12204 tablet_bootstrap.cc:492] T 85a53876196647fb99f14159e5dbbdd7 P 62e9262d25ee40f39310c1b09323a676: No bootstrap required, opened a new log
I20260927 22:11:00.745129 12204 ts_tablet_manager.cc:1397] T 85a53876196647fb99f14159e5dbbdd7 P 62e9262d25ee40f39310c1b09323a676: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260927 22:11:00.746520 12205 tablet_bootstrap.cc:492] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7: No bootstrap required, opened a new log
I20260927 22:11:00.746842 12205 ts_tablet_manager.cc:1397] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 22:11:00.753000 12204 raft_consensus.cc:348] T 85a53876196647fb99f14159e5dbbdd7 P 62e9262d25ee40f39310c1b09323a676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e34aa6a29e5e40b9ad217e8e0d0c1e22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38353 } } peers { permanent_uuid: "62e9262d25ee40f39310c1b09323a676" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32899 } } peers { permanent_uuid: "c6ebb84b014c42e18cfcae21a28d61c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37085 } }
I20260927 22:11:00.753357 12204 raft_consensus.cc:374] T 85a53876196647fb99f14159e5dbbdd7 P 62e9262d25ee40f39310c1b09323a676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:11:00.753482 12204 raft_consensus.cc:729] T 85a53876196647fb99f14159e5dbbdd7 P 62e9262d25ee40f39310c1b09323a676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62e9262d25ee40f39310c1b09323a676, State: Initialized, Role: FOLLOWER
I20260927 22:11:00.753213 12203 raft_consensus.cc:348] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e34aa6a29e5e40b9ad217e8e0d0c1e22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38353 } } peers { permanent_uuid: "62e9262d25ee40f39310c1b09323a676" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32899 } } peers { permanent_uuid: "c6ebb84b014c42e18cfcae21a28d61c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37085 } }
I20260927 22:11:00.753620 12203 raft_consensus.cc:374] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:11:00.753712 12203 raft_consensus.cc:729] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e34aa6a29e5e40b9ad217e8e0d0c1e22, State: Initialized, Role: FOLLOWER
I20260927 22:11:00.754287 12203 consensus_queue.cc:260] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e34aa6a29e5e40b9ad217e8e0d0c1e22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38353 } } peers { permanent_uuid: "62e9262d25ee40f39310c1b09323a676" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32899 } } peers { permanent_uuid: "c6ebb84b014c42e18cfcae21a28d61c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37085 } }
I20260927 22:11:00.754227 12204 consensus_queue.cc:260] T 85a53876196647fb99f14159e5dbbdd7 P 62e9262d25ee40f39310c1b09323a676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e34aa6a29e5e40b9ad217e8e0d0c1e22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38353 } } peers { permanent_uuid: "62e9262d25ee40f39310c1b09323a676" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32899 } } peers { permanent_uuid: "c6ebb84b014c42e18cfcae21a28d61c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37085 } }
I20260927 22:11:00.755789 12050 heartbeater.cc:503] Master 127.0.97.62:42551 was elected leader, sending a full tablet report...
I20260927 22:11:00.756527 12204 ts_tablet_manager.cc:1428] T 85a53876196647fb99f14159e5dbbdd7 P 62e9262d25ee40f39310c1b09323a676: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260927 22:11:00.757721 12205 raft_consensus.cc:348] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e34aa6a29e5e40b9ad217e8e0d0c1e22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38353 } } peers { permanent_uuid: "62e9262d25ee40f39310c1b09323a676" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32899 } } peers { permanent_uuid: "c6ebb84b014c42e18cfcae21a28d61c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37085 } }
I20260927 22:11:00.758176 12205 raft_consensus.cc:374] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:11:00.758327 12205 raft_consensus.cc:729] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ebb84b014c42e18cfcae21a28d61c7, State: Initialized, Role: FOLLOWER
I20260927 22:11:00.759063 12205 consensus_queue.cc:260] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e34aa6a29e5e40b9ad217e8e0d0c1e22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38353 } } peers { permanent_uuid: "62e9262d25ee40f39310c1b09323a676" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32899 } } peers { permanent_uuid: "c6ebb84b014c42e18cfcae21a28d61c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37085 } }
I20260927 22:11:00.760516 12203 ts_tablet_manager.cc:1428] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22: Time spent starting tablet: real 0.018s user 0.012s sys 0.006s
I20260927 22:11:00.760514 12184 heartbeater.cc:503] Master 127.0.97.62:42551 was elected leader, sending a full tablet report...
I20260927 22:11:00.762300 12205 ts_tablet_manager.cc:1428] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7: Time spent starting tablet: real 0.015s user 0.008s sys 0.005s
W20260927 22:11:00.785578 12051 tablet.cc:2369] T 85a53876196647fb99f14159e5dbbdd7 P 62e9262d25ee40f39310c1b09323a676: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:11:00.831909 12209 raft_consensus.cc:482] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:11:00.832264 12209 raft_consensus.cc:504] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e34aa6a29e5e40b9ad217e8e0d0c1e22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38353 } } peers { permanent_uuid: "62e9262d25ee40f39310c1b09323a676" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32899 } } peers { permanent_uuid: "c6ebb84b014c42e18cfcae21a28d61c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37085 } }
I20260927 22:11:00.833314 12211 raft_consensus.cc:482] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:11:00.833630 12211 raft_consensus.cc:504] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e34aa6a29e5e40b9ad217e8e0d0c1e22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38353 } } peers { permanent_uuid: "62e9262d25ee40f39310c1b09323a676" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32899 } } peers { permanent_uuid: "c6ebb84b014c42e18cfcae21a28d61c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37085 } }
I20260927 22:11:00.834692 12209 leader_election.cc:288] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62e9262d25ee40f39310c1b09323a676 (127.0.97.2:32899), c6ebb84b014c42e18cfcae21a28d61c7 (127.0.97.3:37085)
I20260927 22:11:00.836058 12211 leader_election.cc:288] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e34aa6a29e5e40b9ad217e8e0d0c1e22 (127.0.97.1:38353), 62e9262d25ee40f39310c1b09323a676 (127.0.97.2:32899)
I20260927 22:11:00.847054 12187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:00.836357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38353 (local address 127.0.97.3:56665)
0927 22:11:00.836531 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:00.836550 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:00.836637 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:00.837437 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:00.837447 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:00.837502 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:11:00.837792 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:00.837809 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.839865 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.839875 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:00.842904 (+ 3029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:00.842931 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.845810 (+ 2879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.845820 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:00.846175 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:00.846239 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 22:11:00.846365 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 22:11:00.846618 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 22:11:00.847926 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:00.836650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56665 (local address 127.0.97.1:38353)
0927 22:11:00.836857 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:00.836865 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:00.836886 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:11:00.836952 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:00.836958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:00.837043 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:00.837189 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:11:00.838015 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.839639 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.843280 (+ 3641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.845300 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.845505 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:00.845681 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:00.847446 (+ 1765us) server_negotiation.cc:299] Negotiation successful
0927 22:11:00.847613 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:11:00.849334 11862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a53876196647fb99f14159e5dbbdd7" candidate_uuid: "c6ebb84b014c42e18cfcae21a28d61c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e34aa6a29e5e40b9ad217e8e0d0c1e22" is_pre_election: true
I20260927 22:11:00.849890 11862 raft_consensus.cc:2454] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ebb84b014c42e18cfcae21a28d61c7 in term 0.
I20260927 22:11:00.851152 12075 leader_election.cc:302] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ebb84b014c42e18cfcae21a28d61c7, e34aa6a29e5e40b9ad217e8e0d0c1e22; no voters:
I20260927 22:11:00.851951 12211 raft_consensus.cc:2781] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:11:00.852118 12211 raft_consensus.cc:482] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:11:00.852309 12211 raft_consensus.cc:3037] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:11:00.855926 12211 raft_consensus.cc:504] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e34aa6a29e5e40b9ad217e8e0d0c1e22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38353 } } peers { permanent_uuid: "62e9262d25ee40f39310c1b09323a676" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32899 } } peers { permanent_uuid: "c6ebb84b014c42e18cfcae21a28d61c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37085 } }
I20260927 22:11:00.856598 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:00.836777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32899 (local address 127.0.97.3:33539)
0927 22:11:00.847328 (+ 10551us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:00.847352 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:00.847498 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:00.848911 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:00.848916 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:00.848940 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:11:00.849222 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:00.849232 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.851355 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.851360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:00.853470 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:00.853483 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.855636 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.855645 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:00.855848 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:00.855926 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 22:11:00.856020 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 22:11:00.856163 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10373,"mutex_wait_us":2,"thread_start_us":10101,"threads_started":1}
I20260927 22:11:00.857177 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:00.836804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41901 (local address 127.0.97.3:37085)
0927 22:11:00.836991 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:00.836996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:00.847648 (+ 10652us) server_negotiation.cc:407] Connection header received
0927 22:11:00.847957 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:00.847965 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:00.848049 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:00.848124 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:11:00.849241 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.850302 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.854369 (+ 4067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.855793 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.855955 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:00.856055 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:00.856730 (+ 675us) server_negotiation.cc:299] Negotiation successful
0927 22:11:00.856866 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":51}
I20260927 22:11:00.857241 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:00.834992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55735 (local address 127.0.97.2:32899)
0927 22:11:00.835196 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:00.835204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:00.836291 (+ 1087us) server_negotiation.cc:407] Connection header received
0927 22:11:00.836605 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:00.836611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:00.836688 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:00.836777 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:11:00.848839 (+ 12062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.849979 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.854314 (+ 4335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.855806 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.855914 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:00.856013 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:00.856802 (+ 789us) server_negotiation.cc:299] Negotiation successful
0927 22:11:00.856950 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":60}
I20260927 22:11:00.857000 12211 leader_election.cc:288] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7 [CANDIDATE]: Term 1 election: Requested vote from peers e34aa6a29e5e40b9ad217e8e0d0c1e22 (127.0.97.1:38353), 62e9262d25ee40f39310c1b09323a676 (127.0.97.2:32899)
I20260927 22:11:00.857621 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:00.836233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37085 (local address 127.0.97.1:41901)
0927 22:11:00.847531 (+ 11298us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:00.847552 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:00.847769 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:00.848496 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:00.848502 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:00.848537 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:11:00.849026 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:00.849039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.850566 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.850571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:00.854195 (+ 3624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:00.854209 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.856272 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.856279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:00.856494 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:00.856535 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 22:11:00.857036 (+ 501us) client_negotiation.cc:240] Negotiation successful
0927 22:11:00.857187 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11161,"thread_start_us":11349,"threads_started":1}
I20260927 22:11:00.857584 11862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a53876196647fb99f14159e5dbbdd7" candidate_uuid: "c6ebb84b014c42e18cfcae21a28d61c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e34aa6a29e5e40b9ad217e8e0d0c1e22"
I20260927 22:11:00.858024 11862 raft_consensus.cc:3037] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:11:00.858136 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:00.835396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32899 (local address 127.0.97.1:55735)
0927 22:11:00.836096 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:00.836118 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:00.836425 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:00.843438 (+ 7013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:00.843446 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:00.843498 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:11:00.848644 (+ 5146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:00.848657 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.850148 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.850153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:00.854068 (+ 3915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:00.854101 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.856422 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.856428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:00.856597 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:00.856639 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 22:11:00.857499 (+ 860us) client_negotiation.cc:240] Negotiation successful
0927 22:11:00.857666 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":4628,"thread_start_us":399,"threads_started":1}
I20260927 22:11:00.858327 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:00.837519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33539 (local address 127.0.97.2:32899)
0927 22:11:00.848377 (+ 10858us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:00.848383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:00.848408 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:11:00.848477 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:00.848482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:00.848560 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:00.848656 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:11:00.849457 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.851166 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.853660 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.855205 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.855346 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:00.855489 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:00.857826 (+ 2337us) server_negotiation.cc:299] Negotiation successful
0927 22:11:00.857992 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":517,"server-negotiator.queue_time_us":10698,"thread_start_us":10477,"threads_started":1}
I20260927 22:11:00.859306 12139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a53876196647fb99f14159e5dbbdd7" candidate_uuid: "e34aa6a29e5e40b9ad217e8e0d0c1e22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ebb84b014c42e18cfcae21a28d61c7" is_pre_election: true
I20260927 22:11:00.859380 12005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a53876196647fb99f14159e5dbbdd7" candidate_uuid: "e34aa6a29e5e40b9ad217e8e0d0c1e22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62e9262d25ee40f39310c1b09323a676" is_pre_election: true
I20260927 22:11:00.859815 12139 raft_consensus.cc:2379] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e34aa6a29e5e40b9ad217e8e0d0c1e22 in current term 1: Already voted for candidate c6ebb84b014c42e18cfcae21a28d61c7 in this term.
I20260927 22:11:00.859745 12004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a53876196647fb99f14159e5dbbdd7" candidate_uuid: "c6ebb84b014c42e18cfcae21a28d61c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62e9262d25ee40f39310c1b09323a676" is_pre_election: true
I20260927 22:11:00.860072 12005 raft_consensus.cc:2454] T 85a53876196647fb99f14159e5dbbdd7 P 62e9262d25ee40f39310c1b09323a676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e34aa6a29e5e40b9ad217e8e0d0c1e22 in term 0.
I20260927 22:11:00.859423 12003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a53876196647fb99f14159e5dbbdd7" candidate_uuid: "c6ebb84b014c42e18cfcae21a28d61c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62e9262d25ee40f39310c1b09323a676"
I20260927 22:11:00.860594 12003 raft_consensus.cc:3037] T 85a53876196647fb99f14159e5dbbdd7 P 62e9262d25ee40f39310c1b09323a676 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:11:00.861204 11798 leader_election.cc:302] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62e9262d25ee40f39310c1b09323a676, e34aa6a29e5e40b9ad217e8e0d0c1e22; no voters: c6ebb84b014c42e18cfcae21a28d61c7
I20260927 22:11:00.861974 11862 raft_consensus.cc:2454] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ebb84b014c42e18cfcae21a28d61c7 in term 1.
I20260927 22:11:00.862133 12209 raft_consensus.cc:2741] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260927 22:11:00.862483 12075 leader_election.cc:302] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ebb84b014c42e18cfcae21a28d61c7, e34aa6a29e5e40b9ad217e8e0d0c1e22; no voters:
I20260927 22:11:00.862767 12211 raft_consensus.cc:2781] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:11:00.863255 12211 raft_consensus.cc:686] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7 [term 1 LEADER]: Becoming Leader. State: Replica: c6ebb84b014c42e18cfcae21a28d61c7, State: Running, Role: LEADER
I20260927 22:11:00.863732 12003 raft_consensus.cc:2454] T 85a53876196647fb99f14159e5dbbdd7 P 62e9262d25ee40f39310c1b09323a676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ebb84b014c42e18cfcae21a28d61c7 in term 1.
I20260927 22:11:00.863765 12211 consensus_queue.cc:237] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e34aa6a29e5e40b9ad217e8e0d0c1e22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38353 } } peers { permanent_uuid: "62e9262d25ee40f39310c1b09323a676" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32899 } } peers { permanent_uuid: "c6ebb84b014c42e18cfcae21a28d61c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37085 } }
I20260927 22:11:00.868389 11693 catalog_manager.cc:5495] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6ebb84b014c42e18cfcae21a28d61c7 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "c6ebb84b014c42e18cfcae21a28d61c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e34aa6a29e5e40b9ad217e8e0d0c1e22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62e9262d25ee40f39310c1b09323a676" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ebb84b014c42e18cfcae21a28d61c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37085 } health_report { overall_health: HEALTHY } } }
W20260927 22:11:00.869131 12185 tablet.cc:2369] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:11:00.902351 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:00.896572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51810 (local address 127.0.97.3:37085)
0927 22:11:00.896743 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:00.896748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:00.896848 (+ 100us) server_negotiation.cc:407] Connection header received
0927 22:11:00.897016 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:00.897020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:00.897122 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:00.897258 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:11:00.897660 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.898454 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.900164 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.900572 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.900679 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:00.901162 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:11:00.901798 (+ 636us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:11:00.901907 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:00.902057 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 22:11:00.902123 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:11:00.910887 12003 raft_consensus.cc:1261] T 85a53876196647fb99f14159e5dbbdd7 P 62e9262d25ee40f39310c1b09323a676 [term 1 FOLLOWER]: Refusing update from remote peer c6ebb84b014c42e18cfcae21a28d61c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:11:00.910962 11862 raft_consensus.cc:1261] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22 [term 1 FOLLOWER]: Refusing update from remote peer c6ebb84b014c42e18cfcae21a28d61c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:11:00.911814 12216 consensus_queue.cc:1035] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62e9262d25ee40f39310c1b09323a676" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:11:00.912235 12211 consensus_queue.cc:1035] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e34aa6a29e5e40b9ad217e8e0d0c1e22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:11:00.919425 11908 tablet.cc:2369] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:11:00.944531 11693 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:11:00.944839 11693 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:11:00.948467 11693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c33ccda1390b454ea286af5d8182d3a2: Sending DeleteTablet for 3 replicas of tablet 85a53876196647fb99f14159e5dbbdd7
I20260927 22:11:00.949642 11842 tablet_service.cc:1514] Processing DeleteTablet for tablet 85a53876196647fb99f14159e5dbbdd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:11:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46440
I20260927 22:11:00.949784 11985 tablet_service.cc:1514] Processing DeleteTablet for tablet 85a53876196647fb99f14159e5dbbdd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:11:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37644
I20260927 22:11:00.950449 12119 tablet_service.cc:1514] Processing DeleteTablet for tablet 85a53876196647fb99f14159e5dbbdd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:11:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51800
I20260927 22:11:00.951670 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 11774
I20260927 22:11:00.962818 12230 tablet_replica.cc:330] T 85a53876196647fb99f14159e5dbbdd7 P 62e9262d25ee40f39310c1b09323a676: stopping tablet replica
I20260927 22:11:00.963382 12230 raft_consensus.cc:2229] T 85a53876196647fb99f14159e5dbbdd7 P 62e9262d25ee40f39310c1b09323a676 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:11:00.964028 12230 raft_consensus.cc:2258] T 85a53876196647fb99f14159e5dbbdd7 P 62e9262d25ee40f39310c1b09323a676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:11:00.966076 12230 ts_tablet_manager.cc:1905] T 85a53876196647fb99f14159e5dbbdd7 P 62e9262d25ee40f39310c1b09323a676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:11:00.973044 12231 tablet_replica.cc:330] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7: stopping tablet replica
I20260927 22:11:00.973579 12230 ts_tablet_manager.cc:1918] T 85a53876196647fb99f14159e5dbbdd7 P 62e9262d25ee40f39310c1b09323a676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:11:00.973668 12229 tablet_replica.cc:330] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22: stopping tablet replica
I20260927 22:11:00.973860 12230 log.cc:1196] T 85a53876196647fb99f14159e5dbbdd7 P 62e9262d25ee40f39310c1b09323a676: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/85a53876196647fb99f14159e5dbbdd7
I20260927 22:11:00.974001 12229 raft_consensus.cc:2229] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:11:00.974436 12230 ts_tablet_manager.cc:1939] T 85a53876196647fb99f14159e5dbbdd7 P 62e9262d25ee40f39310c1b09323a676: Deleting consensus metadata
I20260927 22:11:00.974521 12229 raft_consensus.cc:2258] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:11:00.974958 12231 raft_consensus.cc:2229] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:11:00.975504 12231 raft_consensus.cc:2258] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:11:00.976409 11681 catalog_manager.cc:4841] TS 62e9262d25ee40f39310c1b09323a676 (127.0.97.2:32899): tablet 85a53876196647fb99f14159e5dbbdd7 (table test-table [id=189198890d23485c9b013b9e5823cec5]) successfully deleted
I20260927 22:11:00.978010 12229 ts_tablet_manager.cc:1905] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:11:00.978016 12231 ts_tablet_manager.cc:1905] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 22:11:00 dist-test-slave-mp03 krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:11:00.985320 12231 ts_tablet_manager.cc:1918] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:11:00.985320 12229 ts_tablet_manager.cc:1918] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:11:00.985491 12229 log.cc:1196] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-0/wal/wals/85a53876196647fb99f14159e5dbbdd7
I20260927 22:11:00.985616 12231 log.cc:1196] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/85a53876196647fb99f14159e5dbbdd7
I20260927 22:11:00.986644 12231 ts_tablet_manager.cc:1939] T 85a53876196647fb99f14159e5dbbdd7 P c6ebb84b014c42e18cfcae21a28d61c7: Deleting consensus metadata
I20260927 22:11:00.986660 12229 ts_tablet_manager.cc:1939] T 85a53876196647fb99f14159e5dbbdd7 P e34aa6a29e5e40b9ad217e8e0d0c1e22: Deleting consensus metadata
I20260927 22:11:00.988554 11681 catalog_manager.cc:4841] TS c6ebb84b014c42e18cfcae21a28d61c7 (127.0.97.3:37085): tablet 85a53876196647fb99f14159e5dbbdd7 (table test-table [id=189198890d23485c9b013b9e5823cec5]) successfully deleted
I20260927 22:11:00.988917 11681 catalog_manager.cc:4841] TS e34aa6a29e5e40b9ad217e8e0d0c1e22 (127.0.97.1:38353): tablet 85a53876196647fb99f14159e5dbbdd7 (table test-table [id=189198890d23485c9b013b9e5823cec5]) successfully deleted
I20260927 22:11:00.992322 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:00.953207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46452 (local address 127.0.97.1:38353)
0927 22:11:00.953423 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:00.953432 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:00.963262 (+ 9830us) server_negotiation.cc:407] Connection header received
0927 22:11:00.963785 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:00.963794 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:00.963918 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:00.964006 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:00.966150 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.967554 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.970026 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:00.970938 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:00.971062 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:00.984615 (+ 13553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:00.984631 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:00.984651 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:00.984707 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:00.989184 (+ 4477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:00.990005 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:00.990016 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:00.990020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:00.990178 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:00.990549 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:00.990555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:00.990558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:00.991082 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:00.991369 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:00.991697 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 22:11:00.991924 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:11:00.993052 11902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:11:01.136036 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 11774
I20260927 22:11:01.156008 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 11913
Sep 27 22:11:01 dist-test-slave-mp03 krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:11:01.174199 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:01.156995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37656 (local address 127.0.97.2:32899)
0927 22:11:01.157236 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:01.157242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:01.157264 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:11:01.157336 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:01.157341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:01.157424 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:01.157532 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:01.158976 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:01.160118 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:01.161622 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:01.162228 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:01.162358 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:01.167241 (+ 4883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:01.167262 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:01.167285 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:01.167336 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:01.171179 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:01.172040 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:01.172050 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:01.172053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:01.172208 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:01.172585 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:01.172590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:01.172593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:01.173037 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:01.173330 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:01.173598 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 22:11:01.173836 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:11:01.175024 12045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:11:01.269382 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 11913
I20260927 22:11:01.292032 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 12056
Sep 27 22:11:01 dist-test-slave-mp03 krb5kdc[11218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:11:01.311168 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:01.292934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51820 (local address 127.0.97.3:37085)
0927 22:11:01.293127 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:01.293131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:01.293193 (+ 62us) server_negotiation.cc:407] Connection header received
0927 22:11:01.293425 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:01.293429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:01.293538 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:01.293635 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:01.295425 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:01.296248 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:01.298212 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:01.298903 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:01.299036 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:01.304517 (+ 5481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:01.304533 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:01.304550 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:01.304596 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:01.308055 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:01.308800 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:01.308808 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:01.308810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:01.308948 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:01.309362 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:01.309369 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:01.309373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:01.309807 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:01.310206 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:01.310541 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 22:11:01.310810 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 22:11:01.311962 12179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:11:01.440268 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 12056
I20260927 22:11:01.456058 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 11662
I20260927 22:11:01.457311 11724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:11:01.568921 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 11662
I20260927 22:11:01.593858 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 22:11:01.929467 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 22:11:01.929719 388 mini_ranger.cc:66] Stopping Ranger...
I20260927 22:11:02.275527 388 mini_ranger.cc:68] Stopped Ranger
2026-09-27 22:11:02.275 UTC [11248] LOG: received smart shutdown request
2026-09-27 22:11:02.279 UTC [11248] LOG: background worker "logical replication launcher" (PID 11257) exited with exit code 1
2026-09-27 22:11:02.279 UTC [11252] LOG: shutting down
2026-09-27 22:11:02.461 UTC [11248] LOG: database system is shut down
2026-09-27T22:11:02Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50691 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:11:02 dist-test-slave-mp03 krb5kdc[12245](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:11:02 dist-test-slave-mp03 krb5kdc[12245](info): set up 2 sockets
Sep 27 22:11:02 dist-test-slave-mp03 krb5kdc[12245](info): commencing operation
krb5kdc: starting...
W20260927 22:11:03.715024 388 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:11:03 dist-test-slave-mp03 krb5kdc[12245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:11:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:11:03Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260927 22:11:04.039637 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task3T1nfn/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/postgres -l logfile start
2026-09-27 22:11:06.724 UTC [12275] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 22:11:06.724 UTC [12275] LOG: listening on IPv4 address "127.0.97.20", port 34871
2026-09-27 22:11:06.728 UTC [12275] LOG: listening on Unix socket "/tmp/.s.PGSQL.34871"
2026-09-27 22:11:06.750 UTC [12278] LOG: database system was shut down at 2026-09-27 22:11:05 UTC
2026-09-27 22:11:06.753 UTC [12275] LOG: database system is ready to accept connections
I20260927 22:11:07.750759 388 mini_postgres.cc:96] Postgres bound to 34871
2026-09-27 22:11:07.768 UTC [12286] FATAL: database "slave" does not exist
127.0.97.20:34871 - accepting connections
I20260927 22:11:07.768865 388 mini_ranger.cc:161] Starting Ranger...
I20260927 22:11:07.796289 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 22:11:08.344679 388 mini_ranger.cc:87] Created ranger Postgres database
I20260927 22:11:08.344805 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ranger-admin
2026-09-27 22:11:08,641 [I] DB FLAVOR :POSTGRES
2026-09-27 22:11:08,642 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:11:08,642 [I] Checking connection..
2026-09-27 22:11:08,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 22:11:08,898 [I] Checking connection passed.
2026-09-27 22:11:08,898 [I] --------- Verifying version history table ---------
2026-09-27 22:11:08,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:11:09,141 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:11:09,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:11:09,381 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:11:09,382 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 22:11:09,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34871/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 22:11:09.614 UTC [12350] WARNING: there is no transaction in progress
2026-09-27 22:11:09,621 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 22:11:09,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:11:09,878 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 22:11:09,878 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:11:09,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 22:11:10,116 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 22:11:10,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mp03.c.gcp-upstream.internal','N') ;"
2026-09-27 22:11:10,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:11:10,874 [I] Table x_portal_user does not exist in database ranger
2026-09-27 22:11:10,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:11:11,121 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 22:11:11,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:11:11,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:11:11,618 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 22:11:11,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34871/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 22:11:12.635 UTC [12458] WARNING: there is no transaction in progress
2026-09-27 22:11:12.652 UTC [12458] WARNING: there is no transaction in progress
2026-09-27 22:11:12.670 UTC [12458] WARNING: there is no transaction in progress
2026-09-27 22:11:12.686 UTC [12458] WARNING: there is no transaction in progress
2026-09-27 22:11:12.703 UTC [12458] WARNING: there is no transaction in progress
2026-09-27 22:11:12.723 UTC [12458] WARNING: there is no transaction in progress
2026-09-27 22:11:12.791 UTC [12458] WARNING: there is no transaction in progress
2026-09-27 22:11:12.798 UTC [12458] WARNING: there is no transaction in progress
2026-09-27 22:11:12.814 UTC [12458] WARNING: there is no transaction in progress
2026-09-27 22:11:12.826 UTC [12458] WARNING: there is no transaction in progress
2026-09-27 22:11:12.837 UTC [12458] WARNING: there is no transaction in progress
2026-09-27 22:11:12.853 UTC [12458] WARNING: there is no transaction in progress
2026-09-27 22:11:12.864 UTC [12458] WARNING: there is no transaction in progress
2026-09-27 22:11:12.872 UTC [12458] WARNING: there is no transaction in progress
2026-09-27 22:11:12.880 UTC [12458] WARNING: there is no transaction in progress
2026-09-27 22:11:12.887 UTC [12458] WARNING: there is no transaction in progress
2026-09-27 22:11:13,571 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 22:11:13,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 22:11:13,806 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 22:11:13,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:11:14,081 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 22:11:14,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:11:14,316 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 22:11:14,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:11:14,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:11:14,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mp03.c.gcp-upstream.internal';"
2026-09-27 22:11:15,072 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 22:11:15,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 22:11:15,312 [I] DB_PATCHES have already been applied
I20260927 22:11:15.318581 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ranger-admin:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task3T1nfn/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 22:11:15.318746 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260927 22:11:15.333380 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:43935
Sep 27, 2026 10:11:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:11:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:11:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 10:11:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 10:11:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 10:11:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 10:11:16 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 22:11:16.459561 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.691s user 0.003s sys 0.035s
Sep 27, 2026 10:11:36 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 10:11:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 27, 2026 10:11:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 10:11:36 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 10:11:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 10:11:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 10:11:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 10:11:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 10:11:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 10:11:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 10:11:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 10:11:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 10:11:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 10:11:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 10:11:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 10:11:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 10:11:36 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 10:11:37 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 22:11:38.508404 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 22:11:38 dist-test-slave-mp03 krb5kdc[12245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547098, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 22:11:38.685671 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 22:11:38.707325 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 22:11:39.637179 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 22:11:39.637310 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ranger-kms
I20260927 22:11:39.637356 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:34871
2026-09-27 22:11:39,999 [I] DB FLAVOR :POSTGRES
2026-09-27 22:11:39,999 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:11:39,999 [I] Checking connection
2026-09-27 22:11:39,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34871/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 22:11:40,257 [I] connection success
2026-09-27 22:11:40,258 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 22:11:40,258 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 22:11:40,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34871/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 22:11:40,492 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 22:11:40,492 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:11:40,492 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 22:11:40,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34871/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 22:11:40,769 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 22:11:41.278569 388 mini_ranger_kms.cc:310] Created kms service
I20260927 22:11:41.492812 388 mini_ranger_kms.cc:326] Created kudu user
I20260927 22:11:41.543578 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 22:11:41 dist-test-slave-mp03 krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547098, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:11:41.960726 388 mini_ranger_kms.cc:384] Added ranger policy
I20260927 22:11:41.961043 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ranger-kms:/tmp/dist-test-task3T1nfn/build/asan/bin/postgresql.jar:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task3T1nfn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 22:11:41.961100 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260927 22:11:41.977947 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12638
I20260927 22:11:41.978080 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:41701
Sep 27, 2026 10:11:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:11:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:11:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 10:11:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 10:11:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 10:11:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 22:11:44 dist-test-slave-mp03 krb5kdc[12245](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790547104, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:11:44.282740 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.597s user 0.061s sys 0.318s
I20260927 22:11:44.282998 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 22:11:44.283187 388 mini_ranger_kms.cc:401] 127.0.97.20:41701/kms/v1/keys
Sep 27 22:11:44 dist-test-slave-mp03 krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547104, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 22:11:44 dist-test-slave-mp03 krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547104, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 22:11:47 dist-test-slave-mp03 krb5kdc[12245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:11:47.311677 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46503
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46503
--ranger_config_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41701/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:11:47.650946 12689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:11:47.651295 12689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:11:47.651469 12689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:11:47.658704 12689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:11:47.658844 12689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:11:47.658941 12689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:11:47.659040 12689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:11:47.659132 12689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:11:47.669373 12689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:41701/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46503
--ranger_config_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:11:47.670917 12689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:11:47.672281 12689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:11:47.685719 12694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:11:47.685719 12695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:11:47.686698 12689 server_base.cc:1034] running on GCE node
W20260927 22:11:47.686900 12697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:11:47.687700 12689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:11:47.689116 12689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:11:47.690343 12689 hybrid_clock.cc:648] HybridClock initialized: now 1790547107690287 us; error 50 us; skew 500 ppm
I20260927 22:11:47.690740 12689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:11:47 dist-test-slave-mp03 krb5kdc[12245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:11:47.693976 12689 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:11:47.695552 12689 webserver.cc:460] Webserver started at http://127.0.97.62:33229/ using document root <none> and password file <none>
I20260927 22:11:47.696198 12689 fs_manager.cc:359] Metadata directory not provided
I20260927 22:11:47.696341 12689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:11:47.696638 12689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:11:47 dist-test-slave-mp03 krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:11:47.749115 12689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "94b3e66753774482bc74fddd4c8c16f8"
format_stamp: "Formatted at 2026-09-27 22:11:47 on dist-test-slave-mp03"
server_key: "087e38920b73ac3e96e6ae63b0104ea8"
server_key_iv: "fc28ef571b15bfb4d9a21cd5a8b85dc0"
server_key_version: "kuduclusterkey@0"
I20260927 22:11:47.750065 12689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "94b3e66753774482bc74fddd4c8c16f8"
format_stamp: "Formatted at 2026-09-27 22:11:47 on dist-test-slave-mp03"
server_key: "087e38920b73ac3e96e6ae63b0104ea8"
server_key_iv: "fc28ef571b15bfb4d9a21cd5a8b85dc0"
server_key_version: "kuduclusterkey@0"
I20260927 22:11:47.755887 12689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:11:47.772548 12704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:11:47.774165 12689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:11:47.774348 12689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "94b3e66753774482bc74fddd4c8c16f8"
format_stamp: "Formatted at 2026-09-27 22:11:47 on dist-test-slave-mp03"
server_key: "087e38920b73ac3e96e6ae63b0104ea8"
server_key_iv: "fc28ef571b15bfb4d9a21cd5a8b85dc0"
server_key_version: "kuduclusterkey@0"
I20260927 22:11:47.774572 12689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:11:47.804581 12689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:11:47.808617 12689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:11:47.808939 12689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:11:47.828825 12689 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46503
I20260927 22:11:47.828836 12755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46503 every 8 connection(s)
I20260927 22:11:47.830276 12689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:11:47.834681 12756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:11:47.838255 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 12689
I20260927 22:11:47.838485 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:11:47.839331 388 external_mini_cluster.cc:1428] Setting key 225412b821598614bccc84499a3a6482
I20260927 22:11:47.853974 12756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8: Bootstrap starting.
I20260927 22:11:47.858299 12756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8: Neither blocks nor log segments found. Creating new log.
I20260927 22:11:47.859474 12756 log.cc:824] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8: Log is configured to *not* fsync() on all Append() calls
I20260927 22:11:47.864764 12756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8: No bootstrap required, opened a new log
Sep 27 22:11:47 dist-test-slave-mp03 krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:11:47.874672 12756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b3e66753774482bc74fddd4c8c16f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46503 } }
I20260927 22:11:47.875115 12756 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:11:47.875243 12756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94b3e66753774482bc74fddd4c8c16f8, State: Initialized, Role: FOLLOWER
I20260927 22:11:47.875941 12756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b3e66753774482bc74fddd4c8c16f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46503 } }
I20260927 22:11:47.876199 12756 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:11:47.876355 12756 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:11:47.876555 12756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:11:47.880120 12756 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b3e66753774482bc74fddd4c8c16f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46503 } }
I20260927 22:11:47.880869 12756 leader_election.cc:302] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94b3e66753774482bc74fddd4c8c16f8; no voters:
I20260927 22:11:47.881628 12756 leader_election.cc:288] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:11:47.881747 12761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:11:47.882050 12759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:47.841529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52292 (local address 127.0.97.62:46503)
0927 22:11:47.855315 (+ 13786us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:47.855335 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:47.856389 (+ 1054us) server_negotiation.cc:407] Connection header received
0927 22:11:47.858972 (+ 2583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:47.859004 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:47.859571 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:47.860073 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:47.862087 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:47.864284 (+ 2197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:47.866198 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:47.867360 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:47.867568 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:47.872601 (+ 5033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:47.872617 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:47.872641 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:47.872697 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:47.876721 (+ 4024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:47.877617 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:47.877626 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:47.877635 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:47.877848 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:47.878344 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:47.878350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:47.878353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:47.879197 (+ 844us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:47.879515 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:47.879986 (+ 471us) server_negotiation.cc:299] Negotiation successful
0927 22:11:47.880443 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13531,"thread_start_us":13293,"threads_started":1}
I20260927 22:11:47.887601 12761 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [term 1 LEADER]: Becoming Leader. State: Replica: 94b3e66753774482bc74fddd4c8c16f8, State: Running, Role: LEADER
I20260927 22:11:47.888286 12756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:11:47.888288 12761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b3e66753774482bc74fddd4c8c16f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46503 } }
I20260927 22:11:47.891328 12763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94b3e66753774482bc74fddd4c8c16f8. Latest consensus state: current_term: 1 leader_uuid: "94b3e66753774482bc74fddd4c8c16f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b3e66753774482bc74fddd4c8c16f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46503 } } }
I20260927 22:11:47.891687 12763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:11:47.891992 12762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94b3e66753774482bc74fddd4c8c16f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b3e66753774482bc74fddd4c8c16f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46503 } } }
I20260927 22:11:47.892611 12762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:11:47.893566 12769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:11:47.906670 12769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:11:47.918551 12769 catalog_manager.cc:1347] Generated new cluster ID: 38c7cc534fe64d57b15318687ac6ae8e
I20260927 22:11:47.918720 12769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:11:47.946390 12756 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 22:11:47.968067 12769 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:11:47.970570 12769 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:11:47.982635 12769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8: Generated new TSK 0
I20260927 22:11:47.984241 12769 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:11:48.834206 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46503
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:11:49 dist-test-slave-mp03 krb5kdc[12245](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790547109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 22:11:49.291217 12801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:11:49.291657 12801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:11:49.291864 12801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:11:49.303966 12801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:11:49.304174 12801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:11:49.304378 12801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:11:49.320914 12801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:41701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:11:49.323104 12801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:11:49.325114 12801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:11:49.346486 12811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:11:49.346625 12812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:11:49.360443 12814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:11:49.363032 12801 server_base.cc:1034] running on GCE node
I20260927 22:11:49.363933 12801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:11:49.365262 12801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:11:49.366571 12801 hybrid_clock.cc:648] HybridClock initialized: now 1790547109366515 us; error 57 us; skew 500 ppm
I20260927 22:11:49.367004 12801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:11:49 dist-test-slave-mp03 krb5kdc[12245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:11:49.370683 12801 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:11:49.374939 12801 webserver.cc:460] Webserver started at http://127.0.97.1:43655/ using document root <none> and password file <none>
I20260927 22:11:49.375672 12801 fs_manager.cc:359] Metadata directory not provided
I20260927 22:11:49.375811 12801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:11:49.376123 12801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:11:49 dist-test-slave-mp03 krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:11:49.398155 12801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "aa49cc638989433f97e90de0bfb8b4d8"
format_stamp: "Formatted at 2026-09-27 22:11:49 on dist-test-slave-mp03"
server_key: "77169e0dc46a95a45a0d1d9c288428c8"
server_key_iv: "85782b58b2a03d99a3864b09f6fa83bf"
server_key_version: "kuduclusterkey@0"
I20260927 22:11:49.399267 12801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "aa49cc638989433f97e90de0bfb8b4d8"
format_stamp: "Formatted at 2026-09-27 22:11:49 on dist-test-slave-mp03"
server_key: "77169e0dc46a95a45a0d1d9c288428c8"
server_key_iv: "85782b58b2a03d99a3864b09f6fa83bf"
server_key_version: "kuduclusterkey@0"
I20260927 22:11:49.413941 12801 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.004s sys 0.008s
I20260927 22:11:49.455232 12821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:11:49.457336 12801 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.006s sys 0.013s
I20260927 22:11:49.457561 12801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "aa49cc638989433f97e90de0bfb8b4d8"
format_stamp: "Formatted at 2026-09-27 22:11:49 on dist-test-slave-mp03"
server_key: "77169e0dc46a95a45a0d1d9c288428c8"
server_key_iv: "85782b58b2a03d99a3864b09f6fa83bf"
server_key_version: "kuduclusterkey@0"
I20260927 22:11:49.457808 12801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:11:49.503389 12801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:11:49.508975 12801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:11:49.509480 12801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:11:49.523065 12801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:11:49.535524 12801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:11:49.535723 12801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:11:49.535945 12801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:11:49.536058 12801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:11:49 dist-test-slave-mp03 krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:11:49.772444 12801 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33593
I20260927 22:11:49.772446 12933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33593 every 8 connection(s)
I20260927 22:11:49.773878 12801 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:11:49.783087 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 12801
I20260927 22:11:49.783341 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:11:49.784180 388 external_mini_cluster.cc:1428] Setting key 5d3cb427ee40bf8e702737b602ae02e2
Sep 27 22:11:49 dist-test-slave-mp03 krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:11:49.808759 12937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:49.779284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46503 (local address 127.0.97.1:43663)
0927 22:11:49.782576 (+ 3292us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:49.782648 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:49.784745 (+ 2097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:49.786175 (+ 1430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:49.786188 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:49.786849 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:11:49.787736 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:49.787768 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:49.789557 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:49.789565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:49.792077 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:49.792097 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:49.793557 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:49.793566 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:49.793944 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:49.795450 (+ 1506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:11:49.795486 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:11:49.798562 (+ 3076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:11:49.803327 (+ 4765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:49.803340 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:49.803364 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:49.804051 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:49.804530 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:49.804537 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:49.804543 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:49.804820 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:49.805639 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:11:49.805650 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:11:49.806159 (+ 509us) client_negotiation.cc:769] Sending connection context
0927 22:11:49.806485 (+ 326us) client_negotiation.cc:240] Negotiation successful
0927 22:11:49.807146 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2981,"thread_start_us":2749,"threads_started":1}
I20260927 22:11:49.809965 12936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:49.778997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43663 (local address 127.0.97.62:46503)
0927 22:11:49.783586 (+ 4589us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:49.783593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:49.783620 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:11:49.785542 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:49.785547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:49.785642 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:49.785737 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:49.787980 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:49.789351 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:49.792289 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:49.793067 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:49.793211 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:49.798926 (+ 5715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:49.798936 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:49.798944 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:49.799009 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:49.803130 (+ 4121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:49.804211 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:49.804220 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:49.804227 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:49.804391 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:49.804955 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:49.804963 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:49.804969 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:49.805414 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:49.805566 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:49.809213 (+ 3647us) server_negotiation.cc:299] Negotiation successful
0927 22:11:49.809505 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4424,"thread_start_us":4172,"threads_started":1}
I20260927 22:11:49.812357 12934 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46503
I20260927 22:11:49.812685 12934 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:11:49.813537 12934 heartbeater.cc:511] Master 127.0.97.62:46503 requested a full tablet report, sending...
I20260927 22:11:49.816214 12721 ts_manager.cc:194] Registered new tserver with Master: aa49cc638989433f97e90de0bfb8b4d8 (127.0.97.1:33593)
I20260927 22:11:49.817698 12721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:11:49.850533 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46503
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:11:49 dist-test-slave-mp03 krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260927 22:11:50.277568 12940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:11:50.277928 12940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:11:50.278087 12940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:11:50.285501 12940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:11:50.285651 12940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:11:50.285830 12940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:11:50.295861 12940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:41701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:11:50.297209 12940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:11:50.298480 12940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:11:50.311983 12954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:11:50.312040 12955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:11:50.312124 12957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:11:50.312932 12940 server_base.cc:1034] running on GCE node
I20260927 22:11:50.313568 12940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:11:50.314532 12940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:11:50.315788 12940 hybrid_clock.cc:648] HybridClock initialized: now 1790547110315741 us; error 37 us; skew 500 ppm
I20260927 22:11:50.316125 12940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:11:50 dist-test-slave-mp03 krb5kdc[12245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:11:50.319418 12940 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:11:50.320807 12940 webserver.cc:460] Webserver started at http://127.0.97.2:44235/ using document root <none> and password file <none>
I20260927 22:11:50.321470 12940 fs_manager.cc:359] Metadata directory not provided
I20260927 22:11:50.321568 12940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:11:50.321816 12940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:11:50 dist-test-slave-mp03 krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:11:50.337898 12940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "09f92d371a164ab588cc8f48b92aa597"
format_stamp: "Formatted at 2026-09-27 22:11:50 on dist-test-slave-mp03"
server_key: "9f07fef5d8cda50884db80f5c2c792ef"
server_key_iv: "2ab253ada404aea6a174ed1dd10bdc36"
server_key_version: "kuduclusterkey@0"
I20260927 22:11:50.338840 12940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "09f92d371a164ab588cc8f48b92aa597"
format_stamp: "Formatted at 2026-09-27 22:11:50 on dist-test-slave-mp03"
server_key: "9f07fef5d8cda50884db80f5c2c792ef"
server_key_iv: "2ab253ada404aea6a174ed1dd10bdc36"
server_key_version: "kuduclusterkey@0"
I20260927 22:11:50.344354 12940 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:11:50.360924 12964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:11:50.362742 12940 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 22:11:50.362939 12940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "09f92d371a164ab588cc8f48b92aa597"
format_stamp: "Formatted at 2026-09-27 22:11:50 on dist-test-slave-mp03"
server_key: "9f07fef5d8cda50884db80f5c2c792ef"
server_key_iv: "2ab253ada404aea6a174ed1dd10bdc36"
server_key_version: "kuduclusterkey@0"
I20260927 22:11:50.363175 12940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:11:50.390163 12940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:11:50.395335 12940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:11:50.395709 12940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:11:50.396781 12940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:11:50.398350 12940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:11:50.398459 12940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:11:50.398653 12940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:11:50.398741 12940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 22:11:50.441481 12940 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35041
I20260927 22:11:50.441505 13076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35041 every 8 connection(s)
I20260927 22:11:50.442994 12940 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:11:50.451658 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 12940
I20260927 22:11:50.451848 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:11:50.452289 388 external_mini_cluster.cc:1428] Setting key b52dd4dff2e78f22aef1aadfe8edb8c5
Sep 27 22:11:50 dist-test-slave-mp03 krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:11:50.470472 13079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:50.446287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52183 (local address 127.0.97.62:46503)
0927 22:11:50.446830 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:50.446836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:50.447964 (+ 1128us) server_negotiation.cc:407] Connection header received
0927 22:11:50.450743 (+ 2779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:50.450748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:50.450815 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:50.450911 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:50.453044 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:50.453886 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:50.456561 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:50.457328 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:50.457489 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:50.462075 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:50.462083 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:50.462088 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:50.462144 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:50.466003 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:50.467235 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:50.467244 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:50.467250 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:50.467424 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:50.468021 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:50.468029 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:50.468035 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:50.468459 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:50.468610 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:50.469641 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0927 22:11:50.469945 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":295,"threads_started":1}
I20260927 22:11:50.471748 13080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:50.446826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46503 (local address 127.0.97.2:52183)
0927 22:11:50.447755 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:50.447805 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:50.450404 (+ 2599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:50.451232 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:50.451245 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:50.451939 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:11:50.452829 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:50.452857 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:50.454075 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:50.454082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:50.456397 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:50.456412 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:50.457584 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:50.457592 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:50.457927 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:50.458825 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:11:50.458858 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:11:50.461814 (+ 2956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:11:50.466189 (+ 4375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:50.466200 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:50.466220 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:50.466976 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:50.467560 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:50.467566 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:50.467570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:50.467867 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:50.468603 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:11:50.468613 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:11:50.469135 (+ 522us) client_negotiation.cc:769] Sending connection context
0927 22:11:50.469494 (+ 359us) client_negotiation.cc:240] Negotiation successful
0927 22:11:50.470124 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":357,"threads_started":1}
I20260927 22:11:50.473732 13077 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46503
I20260927 22:11:50.474085 13077 heartbeater.cc:464] Registering TS with master...
I20260927 22:11:50.475032 13077 heartbeater.cc:511] Master 127.0.97.62:46503 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:11:50.476778 12721 ts_manager.cc:194] Registered new tserver with Master: 09f92d371a164ab588cc8f48b92aa597 (127.0.97.2:35041)
I20260927 22:11:50.478040 12721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:11:50.506009 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46503
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:11:50.824687 12934 heartbeater.cc:503] Master 127.0.97.62:46503 was elected leader, sending a full tablet report...
W20260927 22:11:50.851503 13083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:11:50.851800 13083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:11:50.851989 13083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:11:50.859464 13083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:11:50.859611 13083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:11:50.859791 13083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:11:50.871549 13083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:41701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:11:50.872982 13083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:11:50.874253 13083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:11:50.887722 13088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:11:50.887717 13089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:11:50.888202 13091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:11:50.888511 13083 server_base.cc:1034] running on GCE node
I20260927 22:11:50.889216 13083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:11:50.890165 13083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:11:50.891436 13083 hybrid_clock.cc:648] HybridClock initialized: now 1790547110891379 us; error 45 us; skew 500 ppm
I20260927 22:11:50.891824 13083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:11:50 dist-test-slave-mp03 krb5kdc[12245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:11:50.895372 13083 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:11:50.896971 13083 webserver.cc:460] Webserver started at http://127.0.97.3:46385/ using document root <none> and password file <none>
I20260927 22:11:50.897689 13083 fs_manager.cc:359] Metadata directory not provided
I20260927 22:11:50.897840 13083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:11:50.898164 13083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:11:50 dist-test-slave-mp03 krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:11:50.916448 13083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "faec6ca40636465395d66030c4c72c68"
format_stamp: "Formatted at 2026-09-27 22:11:50 on dist-test-slave-mp03"
server_key: "30daaa173918188730c740220529d5af"
server_key_iv: "4e374602033212926f33603ebd79c350"
server_key_version: "kuduclusterkey@0"
I20260927 22:11:50.917416 13083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "faec6ca40636465395d66030c4c72c68"
format_stamp: "Formatted at 2026-09-27 22:11:50 on dist-test-slave-mp03"
server_key: "30daaa173918188730c740220529d5af"
server_key_iv: "4e374602033212926f33603ebd79c350"
server_key_version: "kuduclusterkey@0"
I20260927 22:11:50.923410 13083 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 22:11:50.939463 13098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:11:50.941232 13083 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:11:50.941452 13083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "faec6ca40636465395d66030c4c72c68"
format_stamp: "Formatted at 2026-09-27 22:11:50 on dist-test-slave-mp03"
server_key: "30daaa173918188730c740220529d5af"
server_key_iv: "4e374602033212926f33603ebd79c350"
server_key_version: "kuduclusterkey@0"
I20260927 22:11:50.941712 13083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:11:50.961179 13083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:11:50.965327 13083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:11:50.965642 13083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:11:50.966713 13083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:11:50.968210 13083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:11:50.968318 13083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:11:50.968485 13083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:11:50.968560 13083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:11:51.004952 13083 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41053
I20260927 22:11:51.004987 13210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41053 every 8 connection(s)
I20260927 22:11:51.006325 13083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:11:51.012027 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 13083
I20260927 22:11:51.012224 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:11:51.012751 388 external_mini_cluster.cc:1428] Setting key 1af0803d133232ad1aed6a082f03ff85
Sep 27 22:11:51 dist-test-slave-mp03 krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:11:51.030237 13213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:51.009114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59433 (local address 127.0.97.62:46503)
0927 22:11:51.009575 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:51.009580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:51.010561 (+ 981us) server_negotiation.cc:407] Connection header received
0927 22:11:51.012364 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:51.012369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:51.012473 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:51.012588 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:51.014010 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:51.014844 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:51.016337 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:51.016820 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:51.016950 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:51.024312 (+ 7362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:51.024319 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:51.024322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:51.024357 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:51.026949 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:51.027934 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:51.027940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:51.027942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:51.028133 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:51.028657 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:51.028660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:51.028662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:51.028943 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:51.029058 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:51.029737 (+ 679us) server_negotiation.cc:299] Negotiation successful
0927 22:11:51.029986 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":246,"threads_started":1}
I20260927 22:11:51.031312 13214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:51.009650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46503 (local address 127.0.97.3:59433)
0927 22:11:51.010394 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:51.010435 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:51.012077 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:51.012826 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:51.012835 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:51.013277 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:11:51.013848 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:51.013864 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:51.014983 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:51.014989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:51.016204 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:51.016214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:51.017052 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:51.017061 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:51.017289 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:51.021911 (+ 4622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:11:51.021935 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:11:51.024076 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:11:51.027106 (+ 3030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:51.027116 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:51.027135 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:51.027776 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:51.028264 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:51.028268 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:51.028270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:51.028560 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:51.029055 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:11:51.029061 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:11:51.029395 (+ 334us) client_negotiation.cc:769] Sending connection context
0927 22:11:51.029647 (+ 252us) client_negotiation.cc:240] Negotiation successful
0927 22:11:51.030062 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":322,"threads_started":1}
I20260927 22:11:51.032796 13211 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46503
I20260927 22:11:51.033267 13211 heartbeater.cc:464] Registering TS with master...
I20260927 22:11:51.034040 13211 heartbeater.cc:511] Master 127.0.97.62:46503 requested a full tablet report, sending...
I20260927 22:11:51.035524 12721 ts_manager.cc:194] Registered new tserver with Master: faec6ca40636465395d66030c4c72c68 (127.0.97.3:41053)
I20260927 22:11:51.036335 12721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59433
I20260927 22:11:51.040601 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:11:51.042205 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 12801
Sep 27 22:11:51 dist-test-slave-mp03 krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:11:51.058867 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:51.043191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35474 (local address 127.0.97.1:33593)
0927 22:11:51.043756 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:51.043763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:51.043802 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:11:51.043855 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:51.043861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:51.044063 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:51.044181 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:51.045710 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:51.047003 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:51.048477 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:51.048998 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:51.049092 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:51.053416 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:51.053427 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:51.053444 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:51.053478 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:51.056221 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:51.056874 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:51.056881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:51.056883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:51.057054 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:51.057450 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:51.057454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:51.057458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:51.057765 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:51.058022 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:51.058267 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 22:11:51.058455 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":278,"threads_started":1}
I20260927 22:11:51.060259 12929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:11:51.146195 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 12801
I20260927 22:11:51.162237 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 12940
Sep 27 22:11:51 dist-test-slave-mp03 krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:11:51.178777 13218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:51.163169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34466 (local address 127.0.97.2:35041)
0927 22:11:51.163698 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:51.163706 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:51.163732 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:11:51.163792 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:51.163798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:51.164017 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:51.164152 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:51.165315 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:51.166314 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:51.167759 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:51.168341 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:51.168435 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:51.172773 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:51.172786 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:51.172805 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:51.172849 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:51.176042 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:51.176691 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:51.176699 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:51.176701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:51.176841 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:51.177201 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:51.177205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:51.177207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:51.177550 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:51.177777 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:51.178086 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 22:11:51.178432 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":276,"threads_started":1}
I20260927 22:11:51.180263 13072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:11:51.269376 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 12940
I20260927 22:11:51.286262 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 13083
Sep 27 22:11:51 dist-test-slave-mp03 krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:11:51.303009 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:51.287308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52774 (local address 127.0.97.3:41053)
0927 22:11:51.287915 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:51.287921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:51.287956 (+ 35us) server_negotiation.cc:407] Connection header received
0927 22:11:51.288012 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:51.288018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:51.288228 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:51.288395 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:51.289739 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:51.290874 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:51.292163 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:51.292722 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:51.292853 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:51.297218 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:51.297229 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:51.297245 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:51.297282 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:51.300349 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:51.300978 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:51.300990 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:51.300992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:51.301115 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:51.301457 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:51.301464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:51.301469 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:51.301796 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:51.302119 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:51.302372 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 22:11:51.302656 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":325,"threads_started":1}
I20260927 22:11:51.304345 13206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:11:51.387986 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 13083
I20260927 22:11:51.404317 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 12689
I20260927 22:11:51.405788 12751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:11:51.513476 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 12689
I20260927 22:11:51.534384 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46503
--webserver_interface=127.0.97.62
--webserver_port=33229
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46503
--ranger_config_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41701/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:41701/kms with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:11:51.859850 13223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:11:51.860165 13223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:11:51.860349 13223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:11:51.869033 13223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:11:51.869163 13223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:11:51.869272 13223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:11:51.869364 13223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:11:51.869432 13223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:11:51.880743 13223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:41701/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46503
--ranger_config_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=33229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:11:51.882091 13223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:11:51.883584 13223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:11:51.895731 13228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:11:51.895808 13231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:11:51.895776 13229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:11:51.896463 13223 server_base.cc:1034] running on GCE node
I20260927 22:11:51.897215 13223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:11:51.898659 13223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:11:51.899891 13223 hybrid_clock.cc:648] HybridClock initialized: now 1790547111899850 us; error 31 us; skew 500 ppm
I20260927 22:11:51.900236 13223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:11:51 dist-test-slave-mp03 krb5kdc[12245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:11:51.904047 13223 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:11:51.905663 13223 webserver.cc:460] Webserver started at http://127.0.97.62:33229/ using document root <none> and password file <none>
I20260927 22:11:51.906337 13223 fs_manager.cc:359] Metadata directory not provided
I20260927 22:11:51.906479 13223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:11:51 dist-test-slave-mp03 krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:11:51.976341 13223 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 22:11:51.979941 13239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:11:51.981375 13223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:11:51.981530 13223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "94b3e66753774482bc74fddd4c8c16f8"
format_stamp: "Formatted at 2026-09-27 22:11:47 on dist-test-slave-mp03"
server_key: "087e38920b73ac3e96e6ae63b0104ea8"
server_key_iv: "fc28ef571b15bfb4d9a21cd5a8b85dc0"
server_key_version: "kuduclusterkey@0"
I20260927 22:11:51.982486 13223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:11:52.024024 13223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:11:52.028018 13223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:11:52.028254 13223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:11:52.047066 13223 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46503
I20260927 22:11:52.047086 13290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46503 every 8 connection(s)
I20260927 22:11:52.048602 13223 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:11:52.055470 13291 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:11:52.058225 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 13223
I20260927 22:11:52.058768 13291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8: Bootstrap starting.
I20260927 22:11:52.059188 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33593
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46503
--webserver_port=43655
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:41701/kms with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:11:52.077490 13291 log.cc:824] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8: Log is configured to *not* fsync() on all Append() calls
I20260927 22:11:52.086796 13291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:11:52.087420 13291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8: Bootstrap complete.
I20260927 22:11:52.096594 13291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b3e66753774482bc74fddd4c8c16f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46503 } }
I20260927 22:11:52.097440 13291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94b3e66753774482bc74fddd4c8c16f8, State: Initialized, Role: FOLLOWER
I20260927 22:11:52.098073 13291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b3e66753774482bc74fddd4c8c16f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46503 } }
I20260927 22:11:52.098286 13291 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:11:52.098393 13291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:11:52.098583 13291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:11:52.102265 13291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b3e66753774482bc74fddd4c8c16f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46503 } }
I20260927 22:11:52.102805 13291 leader_election.cc:302] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94b3e66753774482bc74fddd4c8c16f8; no voters:
I20260927 22:11:52.103447 13291 leader_election.cc:288] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:11:52.103583 13295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:11:52.104652 13295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [term 2 LEADER]: Becoming Leader. State: Replica: 94b3e66753774482bc74fddd4c8c16f8, State: Running, Role: LEADER
I20260927 22:11:52.105288 13295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b3e66753774482bc74fddd4c8c16f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46503 } }
I20260927 22:11:52.105649 13291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:11:52.107777 13297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94b3e66753774482bc74fddd4c8c16f8. Latest consensus state: current_term: 2 leader_uuid: "94b3e66753774482bc74fddd4c8c16f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b3e66753774482bc74fddd4c8c16f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46503 } } }
I20260927 22:11:52.108238 13297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:11:52.108824 13296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "94b3e66753774482bc74fddd4c8c16f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b3e66753774482bc74fddd4c8c16f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46503 } } }
I20260927 22:11:52.109140 13296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:11:52.112818 13301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:11:52.120121 13301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:11:52.121876 13301 catalog_manager.cc:1259] Loaded cluster ID: 38c7cc534fe64d57b15318687ac6ae8e
I20260927 22:11:52.121981 13301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:11:52.127053 13301 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:11:52.128902 13301 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8: Loaded TSK: 0
I20260927 22:11:52.130779 13301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:11:52.147657 13291 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260927 22:11:52.522558 13293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:11:52.523037 13293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:11:52.523238 13293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:11:52.536528 13293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:11:52.536700 13293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:11:52.537038 13293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:11:52.554904 13293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:41701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43655
--tserver_master_addrs=127.0.97.62:46503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:11:52.556911 13293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:11:52.558879 13293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:11:52.599704 13330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:11:52.600380 13329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:11:52.611011 13333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:11:52.611984 13293 server_base.cc:1034] running on GCE node
I20260927 22:11:52.612784 13293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:11:52.617152 13293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:11:52.618371 13293 hybrid_clock.cc:648] HybridClock initialized: now 1790547112618301 us; error 68 us; skew 500 ppm
I20260927 22:11:52.618798 13293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:11:52 dist-test-slave-mp03 krb5kdc[12245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:11:52.624642 13293 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:11:52.663933 13293 webserver.cc:460] Webserver started at http://127.0.97.1:43655/ using document root <none> and password file <none>
I20260927 22:11:52.665169 13293 fs_manager.cc:359] Metadata directory not provided
I20260927 22:11:52.665418 13293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:11:52 dist-test-slave-mp03 krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:11:52.711761 13293 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.004s sys 0.000s
I20260927 22:11:52.721028 13341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:11:52.722894 13293 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.008s
I20260927 22:11:52.723097 13293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "aa49cc638989433f97e90de0bfb8b4d8"
format_stamp: "Formatted at 2026-09-27 22:11:49 on dist-test-slave-mp03"
server_key: "77169e0dc46a95a45a0d1d9c288428c8"
server_key_iv: "85782b58b2a03d99a3864b09f6fa83bf"
server_key_version: "kuduclusterkey@0"
I20260927 22:11:52.724129 13293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:11:52.745639 13293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:11:52.750097 13293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:11:52.750438 13293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:11:52.751574 13293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:11:52.753141 13293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:11:52.753227 13293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:11:52.753363 13293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:11:52.753455 13293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:11:52.805390 13293 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33593
I20260927 22:11:52.806396 13453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33593 every 8 connection(s)
I20260927 22:11:52.807338 13293 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:11:52.814489 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 13293
I20260927 22:11:52.815327 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:35041
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46503
--webserver_port=44235
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:41701/kms with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:11:52 dist-test-slave-mp03 krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:11:52.856781 13456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:52.811276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58965 (local address 127.0.97.62:46503)
0927 22:11:52.828478 (+ 17202us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:52.828498 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:52.828564 (+ 66us) server_negotiation.cc:407] Connection header received
0927 22:11:52.830732 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:52.830758 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:52.831325 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:52.831769 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:52.834409 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:52.836621 (+ 2212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:52.839484 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:52.840714 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:52.840974 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:52.846250 (+ 5276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:52.846265 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:52.846290 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:52.846351 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:52.851014 (+ 4663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:52.852093 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:52.852104 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:52.852113 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:52.852317 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:52.852890 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:52.852895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:52.852899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:52.853518 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:52.853710 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:52.854910 (+ 1200us) server_negotiation.cc:299] Negotiation successful
0927 22:11:52.855359 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16872,"thread_start_us":16405,"threads_started":1}
I20260927 22:11:52.859012 13457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:52.812532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46503 (local address 127.0.97.1:58965)
0927 22:11:52.824343 (+ 11811us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:52.824415 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:52.828546 (+ 4131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:52.832519 (+ 3973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:52.832534 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:52.833260 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:11:52.834117 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:52.834148 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:52.836846 (+ 2698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:52.836945 (+ 99us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:52.839268 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:52.839289 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:52.841438 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:52.841448 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:52.841791 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:52.842688 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:11:52.842732 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:11:52.845843 (+ 3111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:11:52.851221 (+ 5378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:52.851233 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:52.851255 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:52.851913 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:52.852465 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:52.852470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:52.852474 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:52.852756 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:52.853794 (+ 1038us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:11:52.853803 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:11:52.854233 (+ 430us) client_negotiation.cc:769] Sending connection context
0927 22:11:52.854527 (+ 294us) client_negotiation.cc:240] Negotiation successful
0927 22:11:52.857709 (+ 3182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11353,"thread_start_us":13952,"threads_started":1}
I20260927 22:11:52.862777 13454 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46503
I20260927 22:11:52.863128 13454 heartbeater.cc:464] Registering TS with master...
I20260927 22:11:52.864055 13454 heartbeater.cc:511] Master 127.0.97.62:46503 requested a full tablet report, sending...
W20260927 22:11:52.867560 13454 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46503 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 22:11:52.867797 13454 heartbeater.cc:464] Registering TS with master...
I20260927 22:11:52.868111 13454 heartbeater.cc:511] Master 127.0.97.62:46503 requested a full tablet report, sending...
I20260927 22:11:52.869581 13454 heartbeater.cc:464] Registering TS with master...
I20260927 22:11:52.869979 13454 heartbeater.cc:511] Master 127.0.97.62:46503 requested a full tablet report, sending...
W20260927 22:11:52.871397 13454 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:11:53.230435 13458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:11:53.230890 13458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:11:53.231070 13458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:11:53.242684 13458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:11:53.242815 13458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:11:53.243048 13458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:11:53.258986 13458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:41701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:35041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=44235
--tserver_master_addrs=127.0.97.62:46503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:11:53.260901 13458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:11:53.262699 13458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:11:53.281114 13472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:11:53.281993 13473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:11:53.289104 13478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:11:53.289194 13458 server_base.cc:1034] running on GCE node
I20260927 22:11:53.289976 13458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:11:53.297271 13458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:11:53.298641 13458 hybrid_clock.cc:648] HybridClock initialized: now 1790547113298543 us; error 79 us; skew 500 ppm
I20260927 22:11:53.299116 13458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:11:53 dist-test-slave-mp03 krb5kdc[12245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:11:53.303653 13458 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:11:53.316805 13458 webserver.cc:460] Webserver started at http://127.0.97.2:44235/ using document root <none> and password file <none>
I20260927 22:11:53.317855 13458 fs_manager.cc:359] Metadata directory not provided
I20260927 22:11:53.318039 13458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:11:53 dist-test-slave-mp03 krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:11:53.344046 13458 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 22:11:53.363324 13485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:11:53.365516 13458 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.008s sys 0.007s
I20260927 22:11:53.365722 13458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "09f92d371a164ab588cc8f48b92aa597"
format_stamp: "Formatted at 2026-09-27 22:11:50 on dist-test-slave-mp03"
server_key: "9f07fef5d8cda50884db80f5c2c792ef"
server_key_iv: "2ab253ada404aea6a174ed1dd10bdc36"
server_key_version: "kuduclusterkey@0"
I20260927 22:11:53.366987 13458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 27 22:11:53 dist-test-slave-mp03 krb5kdc[12245](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790547113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:11:53.398231 13458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:11:53.403127 13458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:11:53.403415 13458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:11:53.415167 13458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:11:53.417145 13458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:11:53.417268 13458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:11:53.417407 13458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:11:53.417492 13458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:11:53.468559 13458 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35041
I20260927 22:11:53.468591 13598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35041 every 8 connection(s)
I20260927 22:11:53.470263 13458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:11:53.478886 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 13458
I20260927 22:11:53.479735 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:41053
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46503
--webserver_port=46385
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:41701/kms with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:11:53 dist-test-slave-mp03 krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:11:53.504015 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:53.474902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46503 (local address 127.0.97.2:57925)
0927 22:11:53.479002 (+ 4100us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:53.479052 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:53.481439 (+ 2387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:53.482624 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:53.482637 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:53.483263 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:11:53.484113 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:53.484139 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:53.485645 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:53.485650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:53.487761 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:53.487780 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:53.489053 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:53.489060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:53.489354 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:53.490646 (+ 1292us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:11:53.490680 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:11:53.493681 (+ 3001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:11:53.498237 (+ 4556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:53.498249 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:53.498272 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:53.498983 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:53.499445 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:53.499453 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:53.499459 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:53.499736 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:53.500670 (+ 934us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:11:53.500678 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:11:53.501217 (+ 539us) client_negotiation.cc:769] Sending connection context
0927 22:11:53.501638 (+ 421us) client_negotiation.cc:240] Negotiation successful
0927 22:11:53.502309 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3787,"thread_start_us":3552,"threads_started":1}
I20260927 22:11:53.505371 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:53.474404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57925 (local address 127.0.97.62:46503)
0927 22:11:53.478403 (+ 3999us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:53.478412 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:53.479655 (+ 1243us) server_negotiation.cc:407] Connection header received
0927 22:11:53.481923 (+ 2268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:53.481929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:53.482022 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:53.482136 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:53.484320 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:53.485431 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:53.487939 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:53.488570 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:53.488680 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:53.493966 (+ 5286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:53.493976 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:53.493984 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:53.494031 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:53.498062 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:53.499143 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:53.499152 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:53.499159 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:53.499310 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:53.499875 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:53.499883 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:53.499889 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:53.500415 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:53.500582 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:53.504642 (+ 4060us) server_negotiation.cc:299] Negotiation successful
0927 22:11:53.504938 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3793,"thread_start_us":3683,"threads_started":1}
I20260927 22:11:53.506919 13599 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46503
I20260927 22:11:53.507308 13599 heartbeater.cc:464] Registering TS with master...
I20260927 22:11:53.508270 13599 heartbeater.cc:511] Master 127.0.97.62:46503 requested a full tablet report, sending...
I20260927 22:11:53.510583 13256 ts_manager.cc:194] Registered new tserver with Master: 09f92d371a164ab588cc8f48b92aa597 (127.0.97.2:35041)
I20260927 22:11:53.512933 13256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57925
I20260927 22:11:53.871551 13454 heartbeater.cc:464] Registering TS with master...
I20260927 22:11:53.872064 13454 heartbeater.cc:511] Master 127.0.97.62:46503 requested a full tablet report, sending...
Sep 27 22:11:53 dist-test-slave-mp03 krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:11:53.884147 13256 ts_manager.cc:194] Registered new tserver with Master: aa49cc638989433f97e90de0bfb8b4d8 (127.0.97.1:33593)
I20260927 22:11:53.885320 13256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58965
W20260927 22:11:53.917812 13603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:11:53.918393 13603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:11:53.918586 13603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:11:53.930315 13603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:11:53.930483 13603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:11:53.930755 13603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:11:53.946491 13603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:41701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:41053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=46385
--tserver_master_addrs=127.0.97.62:46503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:11:53.948371 13603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:11:53.950155 13603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:11:53.971489 13608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:11:53.972184 13609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:11:53.984408 13611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:11:53.984963 13603 server_base.cc:1034] running on GCE node
I20260927 22:11:53.985838 13603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:11:53.987200 13603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:11:53.988509 13603 hybrid_clock.cc:648] HybridClock initialized: now 1790547113988436 us; error 54 us; skew 500 ppm
I20260927 22:11:53.988955 13603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:11:53 dist-test-slave-mp03 krb5kdc[12245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:11:53.993610 13603 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:11:53.995817 13603 webserver.cc:460] Webserver started at http://127.0.97.3:46385/ using document root <none> and password file <none>
I20260927 22:11:53.996621 13603 fs_manager.cc:359] Metadata directory not provided
I20260927 22:11:53.996798 13603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:11:54 dist-test-slave-mp03 krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:11:54.016038 13603 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 22:11:54.019210 13619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:11:54.020845 13603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:11:54.021049 13603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "faec6ca40636465395d66030c4c72c68"
format_stamp: "Formatted at 2026-09-27 22:11:50 on dist-test-slave-mp03"
server_key: "30daaa173918188730c740220529d5af"
server_key_iv: "4e374602033212926f33603ebd79c350"
server_key_version: "kuduclusterkey@0"
I20260927 22:11:54.022090 13603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:11:54.040227 13603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:11:54.045665 13603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:11:54.046063 13603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:11:54.057369 13603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:11:54.059259 13603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:11:54.059376 13603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:11:54.059538 13603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:11:54.059614 13603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:11:54.105793 13603 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41053
I20260927 22:11:54.105872 13731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41053 every 8 connection(s)
I20260927 22:11:54.107641 13603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:11:54.116976 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 13603
Sep 27 22:11:54 dist-test-slave-mp03 krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:11:54.143473 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:54.118743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37496 (local address 127.0.97.62:46503)
0927 22:11:54.124790 (+ 6047us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:54.124797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:54.125507 (+ 710us) server_negotiation.cc:407] Connection header received
0927 22:11:54.125579 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:54.125586 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:54.125680 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:54.125789 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:54.128359 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.129504 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.131481 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.132034 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.132145 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:54.135713 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:54.135719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:54.135724 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:54.135768 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:54.139284 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:54.140332 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:54.140338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:54.140341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:54.140490 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:54.140980 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:54.140986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:54.140990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:54.141351 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:54.141668 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:54.142804 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0927 22:11:54.143110 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5928,"thread_start_us":5804,"threads_started":1}
I20260927 22:11:54.150470 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:54.110664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38907 (local address 127.0.97.62:46503)
0927 22:11:54.123082 (+ 12418us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:54.123092 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:54.124047 (+ 955us) server_negotiation.cc:407] Connection header received
0927 22:11:54.127031 (+ 2984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:54.127036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:54.127120 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:54.127195 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:54.129917 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.131144 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.133597 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.134270 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.134370 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:54.143036 (+ 8666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:54.143043 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:54.143048 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:54.143086 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:54.146715 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:54.147644 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:54.147650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:54.147653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:54.147792 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:54.148297 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:54.148303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:54.148306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:54.148682 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:54.148838 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:54.149740 (+ 902us) server_negotiation.cc:299] Negotiation successful
0927 22:11:54.150017 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12239,"thread_start_us":13661,"threads_started":1}
I20260927 22:11:54.151635 13735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:54.111264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46503 (local address 127.0.97.3:38907)
0927 22:11:54.123798 (+ 12534us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:54.123879 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:54.126667 (+ 2788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:54.127860 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:54.127874 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:54.128649 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:11:54.129649 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:54.129676 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.131344 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.131350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:54.133423 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:54.133438 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.134745 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.134752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:54.135080 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:54.139400 (+ 4320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:11:54.139437 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:11:54.142462 (+ 3025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:11:54.146886 (+ 4424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:54.146896 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:54.146916 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:54.147506 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:54.147946 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:54.147951 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:54.147954 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:54.148189 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:54.148826 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:11:54.148835 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:11:54.149295 (+ 460us) client_negotiation.cc:769] Sending connection context
0927 22:11:54.149614 (+ 319us) client_negotiation.cc:240] Negotiation successful
0927 22:11:54.150199 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12134,"thread_start_us":11905,"threads_started":1}
I20260927 22:11:54.153340 13732 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46503
I20260927 22:11:54.153739 13732 heartbeater.cc:464] Registering TS with master...
I20260927 22:11:54.154773 13732 heartbeater.cc:511] Master 127.0.97.62:46503 requested a full tablet report, sending...
I20260927 22:11:54.156512 13256 ts_manager.cc:194] Registered new tserver with Master: faec6ca40636465395d66030c4c72c68 (127.0.97.3:41053)
I20260927 22:11:54.157562 13256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38907
I20260927 22:11:54.162859 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:11:54.211771 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:54.186286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37498 (local address 127.0.97.62:46503)
0927 22:11:54.186533 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:54.186539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:54.190734 (+ 4195us) server_negotiation.cc:407] Connection header received
0927 22:11:54.191078 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:54.191085 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:54.191183 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:54.191267 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:54.193561 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.194732 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.196830 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.197356 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.197503 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:54.204769 (+ 7266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:54.204776 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:54.204781 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:54.204829 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:54.208392 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:54.209350 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:54.209355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:54.209358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:54.209502 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:54.210001 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:54.210007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:54.210010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:54.210376 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:54.210652 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:54.211222 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 22:11:54.211450 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
Sep 27 22:11:54 dist-test-slave-mp03 krb5kdc[12245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:11:54.217890 13256 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:11:54.232640 13256 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:11:54.374166 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:54.310000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35486 (local address 127.0.97.1:33593)
0927 22:11:54.349959 (+ 39959us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:54.349969 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:54.352161 (+ 2192us) server_negotiation.cc:407] Connection header received
0927 22:11:54.352315 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:54.352328 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:54.352698 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:54.352957 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:11:54.355330 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.357391 (+ 2061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.365525 (+ 8134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.370131 (+ 4606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.370265 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:54.370519 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:54.373590 (+ 3071us) server_negotiation.cc:299] Negotiation successful
0927 22:11:54.373807 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39751,"thread_start_us":39965,"threads_started":1}
I20260927 22:11:54.376569 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:54.309410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33593 (local address 127.0.0.1:35486)
0927 22:11:54.351783 (+ 42373us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:54.351808 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:54.351991 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:54.353607 (+ 1616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:54.353619 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:54.353669 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:11:54.355067 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:54.355086 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.358715 (+ 3629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.358721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:54.365305 (+ 6584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:54.365323 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.370725 (+ 5402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.370737 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:54.371083 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:54.371167 (+ 84us) client_negotiation.cc:769] Sending connection context
0927 22:11:54.371325 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 22:11:54.371615 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42214,"thread_start_us":40768,"threads_started":1}
I20260927 22:11:54.381471 13389 tablet_service.cc:1467] Processing CreateTablet for tablet c5d47d3c63574026b82220d1be326553 (DEFAULT_TABLE table=test-table [id=0dfece13193c4f9b90852ad9405b56ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:11:54.383132 13389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d47d3c63574026b82220d1be326553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:11:54.389768 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:54.308923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41053 (local address 127.0.0.1:52778)
0927 22:11:54.349043 (+ 40120us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:54.349104 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:54.349291 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:54.366919 (+ 17628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:54.366924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:54.366968 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:11:54.367302 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:54.367313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.380769 (+ 13456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.380774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:54.382889 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:54.382905 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.388994 (+ 6089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.389001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:54.389169 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:54.389207 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 22:11:54.389278 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 22:11:54.389422 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39788,"thread_start_us":44304,"threads_started":1}
I20260927 22:11:54.391443 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:54.309048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52778 (local address 127.0.97.3:41053)
0927 22:11:54.363881 (+ 54833us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:54.363890 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:54.363925 (+ 35us) server_negotiation.cc:407] Connection header received
0927 22:11:54.364040 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:54.364048 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:54.366672 (+ 2624us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:54.367511 (+ 839us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:11:54.372249 (+ 4738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.376313 (+ 4064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.383118 (+ 6805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.384689 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.390435 (+ 5746us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:54.390756 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:54.390840 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 22:11:54.391095 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54553,"thread_start_us":54634,"threads_started":1}
I20260927 22:11:54.393787 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:54.351100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35041 (local address 127.0.0.1:34472)
0927 22:11:54.361383 (+ 10283us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:54.361410 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:54.361584 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:54.378396 (+ 16812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:54.378402 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:54.378433 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:11:54.378775 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:54.378788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.384857 (+ 6069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.384864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:54.387062 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:54.387078 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.392936 (+ 5858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.392958 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:54.393154 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:54.393191 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:11:54.393268 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 22:11:54.393432 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10088,"thread_start_us":13238,"threads_started":1}
I20260927 22:11:54.395219 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:54.351342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34472 (local address 127.0.97.2:35041)
0927 22:11:54.374743 (+ 23401us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:54.374751 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:54.374782 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:11:54.374880 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:54.374889 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:54.375173 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:54.378984 (+ 3811us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:11:54.379195 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.380583 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.387326 (+ 6743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.388721 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.394409 (+ 5688us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:54.394656 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:54.394730 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 22:11:54.394907 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22026,"thread_start_us":21902,"threads_started":1}
I20260927 22:11:54.395797 13667 tablet_service.cc:1467] Processing CreateTablet for tablet c5d47d3c63574026b82220d1be326553 (DEFAULT_TABLE table=test-table [id=0dfece13193c4f9b90852ad9405b56ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:11:54.397081 13667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d47d3c63574026b82220d1be326553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:11:54.402293 13533 tablet_service.cc:1467] Processing CreateTablet for tablet c5d47d3c63574026b82220d1be326553 (DEFAULT_TABLE table=test-table [id=0dfece13193c4f9b90852ad9405b56ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:11:54.403795 13533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d47d3c63574026b82220d1be326553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:11:54.428819 13754 tablet_bootstrap.cc:492] T c5d47d3c63574026b82220d1be326553 P aa49cc638989433f97e90de0bfb8b4d8: Bootstrap starting.
I20260927 22:11:54.436147 13754 tablet_bootstrap.cc:654] T c5d47d3c63574026b82220d1be326553 P aa49cc638989433f97e90de0bfb8b4d8: Neither blocks nor log segments found. Creating new log.
I20260927 22:11:54.440327 13754 log.cc:824] T c5d47d3c63574026b82220d1be326553 P aa49cc638989433f97e90de0bfb8b4d8: Log is configured to *not* fsync() on all Append() calls
I20260927 22:11:54.441371 13756 tablet_bootstrap.cc:492] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597: Bootstrap starting.
I20260927 22:11:54.443445 13755 tablet_bootstrap.cc:492] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68: Bootstrap starting.
I20260927 22:11:54.448704 13755 tablet_bootstrap.cc:654] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68: Neither blocks nor log segments found. Creating new log.
I20260927 22:11:54.450021 13755 log.cc:824] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68: Log is configured to *not* fsync() on all Append() calls
I20260927 22:11:54.454361 13756 tablet_bootstrap.cc:654] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597: Neither blocks nor log segments found. Creating new log.
I20260927 22:11:54.457592 13754 tablet_bootstrap.cc:492] T c5d47d3c63574026b82220d1be326553 P aa49cc638989433f97e90de0bfb8b4d8: No bootstrap required, opened a new log
I20260927 22:11:54.457959 13754 ts_tablet_manager.cc:1397] T c5d47d3c63574026b82220d1be326553 P aa49cc638989433f97e90de0bfb8b4d8: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.000s
I20260927 22:11:54.459602 13755 tablet_bootstrap.cc:492] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68: No bootstrap required, opened a new log
I20260927 22:11:54.459975 13755 ts_tablet_manager.cc:1397] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20260927 22:11:54.464066 13756 log.cc:824] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597: Log is configured to *not* fsync() on all Append() calls
I20260927 22:11:54.475124 13754 raft_consensus.cc:348] T c5d47d3c63574026b82220d1be326553 P aa49cc638989433f97e90de0bfb8b4d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faec6ca40636465395d66030c4c72c68" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41053 } } peers { permanent_uuid: "aa49cc638989433f97e90de0bfb8b4d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "09f92d371a164ab588cc8f48b92aa597" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35041 } }
I20260927 22:11:54.477084 13754 raft_consensus.cc:374] T c5d47d3c63574026b82220d1be326553 P aa49cc638989433f97e90de0bfb8b4d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:11:54.477480 13754 raft_consensus.cc:729] T c5d47d3c63574026b82220d1be326553 P aa49cc638989433f97e90de0bfb8b4d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa49cc638989433f97e90de0bfb8b4d8, State: Initialized, Role: FOLLOWER
I20260927 22:11:54.479111 13754 consensus_queue.cc:260] T c5d47d3c63574026b82220d1be326553 P aa49cc638989433f97e90de0bfb8b4d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faec6ca40636465395d66030c4c72c68" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41053 } } peers { permanent_uuid: "aa49cc638989433f97e90de0bfb8b4d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "09f92d371a164ab588cc8f48b92aa597" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35041 } }
I20260927 22:11:54.484683 13755 raft_consensus.cc:348] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faec6ca40636465395d66030c4c72c68" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41053 } } peers { permanent_uuid: "aa49cc638989433f97e90de0bfb8b4d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "09f92d371a164ab588cc8f48b92aa597" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35041 } }
I20260927 22:11:54.485698 13756 tablet_bootstrap.cc:492] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597: No bootstrap required, opened a new log
I20260927 22:11:54.485870 13755 raft_consensus.cc:374] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:11:54.486027 13755 raft_consensus.cc:729] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faec6ca40636465395d66030c4c72c68, State: Initialized, Role: FOLLOWER
I20260927 22:11:54.486177 13756 ts_tablet_manager.cc:1397] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597: Time spent bootstrapping tablet: real 0.045s user 0.017s sys 0.000s
I20260927 22:11:54.487222 13755 consensus_queue.cc:260] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faec6ca40636465395d66030c4c72c68" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41053 } } peers { permanent_uuid: "aa49cc638989433f97e90de0bfb8b4d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "09f92d371a164ab588cc8f48b92aa597" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35041 } }
I20260927 22:11:54.488576 13454 heartbeater.cc:503] Master 127.0.97.62:46503 was elected leader, sending a full tablet report...
I20260927 22:11:54.492147 13755 ts_tablet_manager.cc:1428] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68: Time spent starting tablet: real 0.032s user 0.009s sys 0.004s
I20260927 22:11:54.492519 13732 heartbeater.cc:503] Master 127.0.97.62:46503 was elected leader, sending a full tablet report...
I20260927 22:11:54.497802 13754 ts_tablet_manager.cc:1428] T c5d47d3c63574026b82220d1be326553 P aa49cc638989433f97e90de0bfb8b4d8: Time spent starting tablet: real 0.040s user 0.011s sys 0.004s
I20260927 22:11:54.506912 13756 raft_consensus.cc:348] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faec6ca40636465395d66030c4c72c68" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41053 } } peers { permanent_uuid: "aa49cc638989433f97e90de0bfb8b4d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "09f92d371a164ab588cc8f48b92aa597" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35041 } }
I20260927 22:11:54.507510 13756 raft_consensus.cc:374] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:11:54.507658 13756 raft_consensus.cc:729] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f92d371a164ab588cc8f48b92aa597, State: Initialized, Role: FOLLOWER
I20260927 22:11:54.508507 13756 consensus_queue.cc:260] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faec6ca40636465395d66030c4c72c68" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41053 } } peers { permanent_uuid: "aa49cc638989433f97e90de0bfb8b4d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "09f92d371a164ab588cc8f48b92aa597" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35041 } }
I20260927 22:11:54.517887 13599 heartbeater.cc:503] Master 127.0.97.62:46503 was elected leader, sending a full tablet report...
I20260927 22:11:54.534556 13756 ts_tablet_manager.cc:1428] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597: Time spent starting tablet: real 0.048s user 0.008s sys 0.025s
W20260927 22:11:54.566145 13455 tablet.cc:2369] T c5d47d3c63574026b82220d1be326553 P aa49cc638989433f97e90de0bfb8b4d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:11:54.571206 13762 raft_consensus.cc:482] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:11:54.571548 13762 raft_consensus.cc:504] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faec6ca40636465395d66030c4c72c68" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41053 } } peers { permanent_uuid: "aa49cc638989433f97e90de0bfb8b4d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "09f92d371a164ab588cc8f48b92aa597" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35041 } }
I20260927 22:11:54.573685 13762 leader_election.cc:288] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers faec6ca40636465395d66030c4c72c68 (127.0.97.3:41053), aa49cc638989433f97e90de0bfb8b4d8 (127.0.97.1:33593)
I20260927 22:11:54.582664 13761 raft_consensus.cc:482] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:11:54.582970 13761 raft_consensus.cc:504] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faec6ca40636465395d66030c4c72c68" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41053 } } peers { permanent_uuid: "aa49cc638989433f97e90de0bfb8b4d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "09f92d371a164ab588cc8f48b92aa597" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35041 } }
I20260927 22:11:54.585156 13761 leader_election.cc:288] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa49cc638989433f97e90de0bfb8b4d8 (127.0.97.1:33593), 09f92d371a164ab588cc8f48b92aa597 (127.0.97.2:35041)
W20260927 22:11:54.610965 13733 tablet.cc:2369] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:11:54.615224 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:54.586995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46831 (local address 127.0.97.2:35041)
0927 22:11:54.587182 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:54.587191 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:54.594344 (+ 7153us) server_negotiation.cc:407] Connection header received
0927 22:11:54.594558 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:54.594567 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:54.594732 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:54.594850 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:11:54.596623 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.598041 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.603809 (+ 5768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.605362 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.605466 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:54.605576 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:54.614726 (+ 9150us) server_negotiation.cc:299] Negotiation successful
0927 22:11:54.614901 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 22:11:54.615785 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:54.574485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56893 (local address 127.0.97.1:33593)
0927 22:11:54.574747 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:54.574757 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:54.602542 (+ 27785us) server_negotiation.cc:407] Connection header received
0927 22:11:54.604221 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:54.604226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:54.604322 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:54.604387 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:11:54.609329 (+ 4942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.610572 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.613126 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.614508 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.614692 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:54.614802 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:54.615358 (+ 556us) server_negotiation.cc:299] Negotiation successful
0927 22:11:54.615536 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:11:54.616328 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:54.586078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35041 (local address 127.0.97.3:46831)
0927 22:11:54.594204 (+ 8126us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:54.594232 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:54.594396 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:54.595102 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:54.595112 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:54.595166 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:11:54.595597 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:54.595617 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.599234 (+ 3617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.599244 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:54.603592 (+ 4348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:54.603608 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.612172 (+ 8564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.612180 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:54.612379 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:54.612420 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 22:11:54.615718 (+ 3298us) client_negotiation.cc:240] Negotiation successful
0927 22:11:54.615896 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8002,"mutex_wait_us":72,"thread_start_us":7837,"threads_started":1}
I20260927 22:11:54.616837 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:54.574485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51751 (local address 127.0.97.3:41053)
0927 22:11:54.574933 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:54.574942 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:54.601945 (+ 27003us) server_negotiation.cc:407] Connection header received
0927 22:11:54.603201 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:54.603208 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:54.603277 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:54.603334 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:11:54.605338 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.606531 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.614279 (+ 7748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.615675 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.615778 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:54.615875 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:54.616482 (+ 607us) server_negotiation.cc:299] Negotiation successful
0927 22:11:54.616614 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299}
I20260927 22:11:54.617403 13553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d47d3c63574026b82220d1be326553" candidate_uuid: "faec6ca40636465395d66030c4c72c68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f92d371a164ab588cc8f48b92aa597" is_pre_election: true
I20260927 22:11:54.617940 13553 raft_consensus.cc:2454] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faec6ca40636465395d66030c4c72c68 in term 0.
I20260927 22:11:54.618747 13623 leader_election.cc:302] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f92d371a164ab588cc8f48b92aa597, faec6ca40636465395d66030c4c72c68; no voters:
I20260927 22:11:54.619251 13761 raft_consensus.cc:2781] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:11:54.619415 13761 raft_consensus.cc:482] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:11:54.619541 13761 raft_consensus.cc:3037] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:11:54.619874 13735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:54.585722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33593 (local address 127.0.97.3:53973)
0927 22:11:54.585892 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:54.585914 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:54.586055 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:54.595382 (+ 9327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:54.595386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:54.595410 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:11:54.595679 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:54.595693 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.599487 (+ 3794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.599494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:54.602142 (+ 2648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:54.602159 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.619053 (+ 16894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.619059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:54.619269 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:54.619308 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 22:11:54.619388 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 22:11:54.619583 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":35}
I20260927 22:11:54.621050 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:54.586667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53973 (local address 127.0.97.1:33593)
0927 22:11:54.593247 (+ 6580us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:54.593258 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:54.593301 (+ 43us) server_negotiation.cc:407] Connection header received
0927 22:11:54.593418 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:54.593427 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:54.593570 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:54.595872 (+ 2302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:11:54.596166 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.598163 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.617443 (+ 19280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.618862 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.620365 (+ 1503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:54.620509 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:54.620564 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 22:11:54.620704 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6372,"thread_start_us":6116,"threads_started":1}
I20260927 22:11:54.621816 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:54.574272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33593 (local address 127.0.97.2:56893)
0927 22:11:54.602432 (+ 28160us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:54.602465 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:54.602879 (+ 414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:54.604611 (+ 1732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:54.604617 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:54.604649 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:11:54.605104 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:54.605117 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.610793 (+ 5676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.610802 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:54.612935 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:54.612950 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.614956 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.614963 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:54.615150 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:54.615183 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:11:54.616181 (+ 998us) client_negotiation.cc:240] Negotiation successful
0927 22:11:54.616349 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28027,"thread_start_us":27843,"threads_started":1}
I20260927 22:11:54.621850 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:54.574017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41053 (local address 127.0.97.2:51751)
0927 22:11:54.601832 (+ 27815us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:54.601860 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:54.603046 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:54.603528 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:54.603534 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:54.603562 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:11:54.604018 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:54.604030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.606805 (+ 2775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.606811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:54.609127 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:54.609142 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.616093 (+ 6951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.616100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:54.616281 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:54.616315 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:11:54.617252 (+ 937us) client_negotiation.cc:240] Negotiation successful
0927 22:11:54.617402 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27668,"thread_start_us":28547,"threads_started":1}
I20260927 22:11:54.622187 13409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d47d3c63574026b82220d1be326553" candidate_uuid: "faec6ca40636465395d66030c4c72c68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa49cc638989433f97e90de0bfb8b4d8" is_pre_election: true
I20260927 22:11:54.622910 13409 raft_consensus.cc:2454] T c5d47d3c63574026b82220d1be326553 P aa49cc638989433f97e90de0bfb8b4d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faec6ca40636465395d66030c4c72c68 in term 0.
I20260927 22:11:54.623091 13687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d47d3c63574026b82220d1be326553" candidate_uuid: "09f92d371a164ab588cc8f48b92aa597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faec6ca40636465395d66030c4c72c68" is_pre_election: true
I20260927 22:11:54.623451 13408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d47d3c63574026b82220d1be326553" candidate_uuid: "09f92d371a164ab588cc8f48b92aa597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa49cc638989433f97e90de0bfb8b4d8" is_pre_election: true
I20260927 22:11:54.623739 13408 raft_consensus.cc:2454] T c5d47d3c63574026b82220d1be326553 P aa49cc638989433f97e90de0bfb8b4d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f92d371a164ab588cc8f48b92aa597 in term 0.
I20260927 22:11:54.623957 13761 raft_consensus.cc:504] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faec6ca40636465395d66030c4c72c68" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41053 } } peers { permanent_uuid: "aa49cc638989433f97e90de0bfb8b4d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "09f92d371a164ab588cc8f48b92aa597" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35041 } }
I20260927 22:11:54.624706 13761 leader_election.cc:288] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68 [CANDIDATE]: Term 1 election: Requested vote from peers aa49cc638989433f97e90de0bfb8b4d8 (127.0.97.1:33593), 09f92d371a164ab588cc8f48b92aa597 (127.0.97.2:35041)
I20260927 22:11:54.625033 13687 raft_consensus.cc:2379] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 09f92d371a164ab588cc8f48b92aa597 in current term 1: Already voted for candidate faec6ca40636465395d66030c4c72c68 in this term.
I20260927 22:11:54.624909 13489 leader_election.cc:302] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f92d371a164ab588cc8f48b92aa597, aa49cc638989433f97e90de0bfb8b4d8; no voters:
I20260927 22:11:54.626124 13762 raft_consensus.cc:2781] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:11:54.626262 13762 raft_consensus.cc:482] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:11:54.626353 13762 raft_consensus.cc:3037] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:11:54.629562 13762 raft_consensus.cc:504] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faec6ca40636465395d66030c4c72c68" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41053 } } peers { permanent_uuid: "aa49cc638989433f97e90de0bfb8b4d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "09f92d371a164ab588cc8f48b92aa597" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35041 } }
I20260927 22:11:54.630657 13408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d47d3c63574026b82220d1be326553" candidate_uuid: "faec6ca40636465395d66030c4c72c68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa49cc638989433f97e90de0bfb8b4d8"
I20260927 22:11:54.630874 13408 raft_consensus.cc:3037] T c5d47d3c63574026b82220d1be326553 P aa49cc638989433f97e90de0bfb8b4d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:11:54.631058 13409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d47d3c63574026b82220d1be326553" candidate_uuid: "09f92d371a164ab588cc8f48b92aa597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa49cc638989433f97e90de0bfb8b4d8"
I20260927 22:11:54.632149 13687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d47d3c63574026b82220d1be326553" candidate_uuid: "09f92d371a164ab588cc8f48b92aa597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faec6ca40636465395d66030c4c72c68"
I20260927 22:11:54.632400 13687 raft_consensus.cc:2379] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 09f92d371a164ab588cc8f48b92aa597 in current term 1: Already voted for candidate faec6ca40636465395d66030c4c72c68 in this term.
I20260927 22:11:54.632740 13762 leader_election.cc:288] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597 [CANDIDATE]: Term 1 election: Requested vote from peers faec6ca40636465395d66030c4c72c68 (127.0.97.3:41053), aa49cc638989433f97e90de0bfb8b4d8 (127.0.97.1:33593)
I20260927 22:11:54.632903 13488 leader_election.cc:302] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09f92d371a164ab588cc8f48b92aa597; no voters: aa49cc638989433f97e90de0bfb8b4d8, faec6ca40636465395d66030c4c72c68
I20260927 22:11:54.633088 13553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d47d3c63574026b82220d1be326553" candidate_uuid: "faec6ca40636465395d66030c4c72c68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f92d371a164ab588cc8f48b92aa597"
I20260927 22:11:54.633386 13553 raft_consensus.cc:2379] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate faec6ca40636465395d66030c4c72c68 in current term 1: Already voted for candidate 09f92d371a164ab588cc8f48b92aa597 in this term.
I20260927 22:11:54.633610 13762 raft_consensus.cc:2726] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 22:11:54.634949 13408 raft_consensus.cc:2454] T c5d47d3c63574026b82220d1be326553 P aa49cc638989433f97e90de0bfb8b4d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faec6ca40636465395d66030c4c72c68 in term 1.
I20260927 22:11:54.635349 13623 leader_election.cc:302] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: aa49cc638989433f97e90de0bfb8b4d8, faec6ca40636465395d66030c4c72c68; no voters: 09f92d371a164ab588cc8f48b92aa597
I20260927 22:11:54.635708 13761 raft_consensus.cc:2781] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:11:54.636243 13761 raft_consensus.cc:686] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68 [term 1 LEADER]: Becoming Leader. State: Replica: faec6ca40636465395d66030c4c72c68, State: Running, Role: LEADER
I20260927 22:11:54.636806 13761 consensus_queue.cc:237] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faec6ca40636465395d66030c4c72c68" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41053 } } peers { permanent_uuid: "aa49cc638989433f97e90de0bfb8b4d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "09f92d371a164ab588cc8f48b92aa597" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35041 } }
I20260927 22:11:54.641158 13255 catalog_manager.cc:5495] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68 reported cstate change: term changed from 0 to 1, leader changed from <none> to faec6ca40636465395d66030c4c72c68 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "faec6ca40636465395d66030c4c72c68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faec6ca40636465395d66030c4c72c68" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa49cc638989433f97e90de0bfb8b4d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09f92d371a164ab588cc8f48b92aa597" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35041 } health_report { overall_health: UNKNOWN } } }
I20260927 22:11:54.702875 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:54.697444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52782 (local address 127.0.97.3:41053)
0927 22:11:54.697649 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:54.697657 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:54.697682 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:11:54.697969 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:54.697973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:54.698063 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:54.698216 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:11:54.698615 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.699361 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.700746 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.701158 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.701287 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:54.701765 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:11:54.702253 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:11:54.702416 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:54.702493 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 22:11:54.702578 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 22:11:54.712376 13553 raft_consensus.cc:1261] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597 [term 1 FOLLOWER]: Refusing update from remote peer faec6ca40636465395d66030c4c72c68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:11:54.712597 13408 raft_consensus.cc:1261] T c5d47d3c63574026b82220d1be326553 P aa49cc638989433f97e90de0bfb8b4d8 [term 1 FOLLOWER]: Refusing update from remote peer faec6ca40636465395d66030c4c72c68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:11:54.713250 13769 consensus_queue.cc:1035] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f92d371a164ab588cc8f48b92aa597" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:11:54.713647 13761 consensus_queue.cc:1035] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa49cc638989433f97e90de0bfb8b4d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:11:54.728755 13600 tablet.cc:2369] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:11:54.744063 13779 mvcc.cc:205] Tried to move back new op lower bound from 7334080981850320896 to 7334080981556695040. Current Snapshot: MvccSnapshot[applied={T|T < 7334080981850320896}]
I20260927 22:11:54.748999 13255 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:11:54.749316 13255 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:11:54.754261 13255 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94b3e66753774482bc74fddd4c8c16f8: Sending DeleteTablet for 3 replicas of tablet c5d47d3c63574026b82220d1be326553
I20260927 22:11:54.755748 13667 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d47d3c63574026b82220d1be326553 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:11:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52778
I20260927 22:11:54.758399 13389 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d47d3c63574026b82220d1be326553 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:11:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35486
I20260927 22:11:54.759263 13533 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d47d3c63574026b82220d1be326553 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:11:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34472
I20260927 22:11:54.769254 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 13293
I20260927 22:11:54.786027 13788 tablet_replica.cc:330] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597: stopping tablet replica
I20260927 22:11:54.787858 13786 tablet_replica.cc:330] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68: stopping tablet replica
I20260927 22:11:54.788553 13786 raft_consensus.cc:2229] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:11:54.794107 13788 raft_consensus.cc:2229] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:11:54.795495 13788 raft_consensus.cc:2258] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:11:54.796494 13786 raft_consensus.cc:2258] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:11:54.817215 13786 ts_tablet_manager.cc:1905] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:11:54.819162 13787 tablet_replica.cc:330] T c5d47d3c63574026b82220d1be326553 P aa49cc638989433f97e90de0bfb8b4d8: stopping tablet replica
I20260927 22:11:54.820456 13788 ts_tablet_manager.cc:1905] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:11:54.820458 13787 raft_consensus.cc:2229] T c5d47d3c63574026b82220d1be326553 P aa49cc638989433f97e90de0bfb8b4d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:11:54.823452 13787 raft_consensus.cc:2258] T c5d47d3c63574026b82220d1be326553 P aa49cc638989433f97e90de0bfb8b4d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:11:54.828130 13787 ts_tablet_manager.cc:1905] T c5d47d3c63574026b82220d1be326553 P aa49cc638989433f97e90de0bfb8b4d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:11:54.838358 13786 ts_tablet_manager.cc:1918] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:11:54.838651 13786 log.cc:1196] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/c5d47d3c63574026b82220d1be326553
I20260927 22:11:54.842104 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:54.770577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35496 (local address 127.0.97.1:33593)
0927 22:11:54.770824 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:54.770832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:54.819823 (+ 48991us) server_negotiation.cc:407] Connection header received
0927 22:11:54.820785 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:54.820791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:54.820904 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:54.820987 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:54.823978 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.825172 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.827993 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:54.828671 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:54.828784 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:54.832923 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:54.832944 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:54.832967 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:54.833021 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:54.837433 (+ 4412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:54.838559 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:54.838571 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:54.838575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:54.839125 (+ 550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:54.839717 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:54.839723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:54.839726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:54.840314 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:54.840674 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:54.841380 (+ 706us) server_negotiation.cc:299] Negotiation successful
0927 22:11:54.841660 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260927 22:11:54.844156 13449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:11:54.842370 13787 ts_tablet_manager.cc:1918] T c5d47d3c63574026b82220d1be326553 P aa49cc638989433f97e90de0bfb8b4d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:11:54.846446 13787 log.cc:1196] T c5d47d3c63574026b82220d1be326553 P aa49cc638989433f97e90de0bfb8b4d8: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-0/wal/wals/c5d47d3c63574026b82220d1be326553
I20260927 22:11:54.847949 13788 ts_tablet_manager.cc:1918] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:11:54.848132 13788 log.cc:1196] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/c5d47d3c63574026b82220d1be326553
I20260927 22:11:54.848690 13788 ts_tablet_manager.cc:1939] T c5d47d3c63574026b82220d1be326553 P 09f92d371a164ab588cc8f48b92aa597: Deleting consensus metadata
I20260927 22:11:54.850845 13243 catalog_manager.cc:4841] TS 09f92d371a164ab588cc8f48b92aa597 (127.0.97.2:35041): tablet c5d47d3c63574026b82220d1be326553 (table test-table [id=0dfece13193c4f9b90852ad9405b56ba]) successfully deleted
I20260927 22:11:54.851091 13786 ts_tablet_manager.cc:1939] T c5d47d3c63574026b82220d1be326553 P faec6ca40636465395d66030c4c72c68: Deleting consensus metadata
I20260927 22:11:54.852990 13242 catalog_manager.cc:4841] TS faec6ca40636465395d66030c4c72c68 (127.0.97.3:41053): tablet c5d47d3c63574026b82220d1be326553 (table test-table [id=0dfece13193c4f9b90852ad9405b56ba]) successfully deleted
I20260927 22:11:54.988862 13787 ts_tablet_manager.cc:1939] T c5d47d3c63574026b82220d1be326553 P aa49cc638989433f97e90de0bfb8b4d8: Deleting consensus metadata
I20260927 22:11:54.991425 13243 catalog_manager.cc:4841] TS aa49cc638989433f97e90de0bfb8b4d8 (127.0.97.1:33593): tablet c5d47d3c63574026b82220d1be326553 (table test-table [id=0dfece13193c4f9b90852ad9405b56ba]) successfully deleted
I20260927 22:11:54.991528 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 13293
I20260927 22:11:55.009311 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 13458
I20260927 22:11:55.025116 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:55.010195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34488 (local address 127.0.97.2:35041)
0927 22:11:55.010398 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:55.010405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:55.010453 (+ 48us) server_negotiation.cc:407] Connection header received
0927 22:11:55.010524 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:55.010529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:55.010652 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:55.010759 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:55.011980 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:55.013100 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:55.014576 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:55.015242 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:55.015367 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:55.018198 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:55.018217 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:55.018237 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:55.018282 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:55.022054 (+ 3772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:55.022821 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:55.022830 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:55.022834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:55.022997 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:55.023432 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:55.023437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:55.023441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:55.023887 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:55.024177 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:55.024498 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 22:11:55.024731 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:11:55.026021 13594 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:11:55.123175 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 13458
I20260927 22:11:55.147493 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 13603
I20260927 22:11:55.164638 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:55.148300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52794 (local address 127.0.97.3:41053)
0927 22:11:55.148475 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:55.148480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:55.148498 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:11:55.148720 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:55.148724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:55.148790 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:55.148883 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:55.150786 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:55.151611 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:55.153532 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:55.154011 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:55.154131 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:55.158033 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:55.158078 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:55.158100 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:55.158152 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:55.161416 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:55.162324 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:55.162336 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:55.162345 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:55.162495 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:55.163104 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:55.163108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:55.163110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:55.163502 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:55.163781 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:55.164140 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 22:11:55.164315 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:11:55.165422 13727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:11:55.285261 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 13603
I20260927 22:11:55.304548 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 13223
I20260927 22:11:55.305814 13286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:11:55.417248 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 13223
I20260927 22:11:55.439476 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 22:11:55.774789 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 22:11:55.774976 388 mini_ranger.cc:66] Stopping Ranger...
I20260927 22:11:56.130653 388 mini_ranger.cc:68] Stopped Ranger
2026-09-27 22:11:56.130 UTC [12275] LOG: received smart shutdown request
2026-09-27 22:11:56.134 UTC [12275] LOG: background worker "logical replication launcher" (PID 12284) exited with exit code 1
2026-09-27 22:11:56.134 UTC [12279] LOG: shutting down
2026-09-27 22:11:56.320 UTC [12275] LOG: database system is shut down
2026-09-27T22:11:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53867 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:11:56 dist-test-slave-mp03 krb5kdc[13802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:11:56 dist-test-slave-mp03 krb5kdc[13802](info): set up 2 sockets
Sep 27 22:11:56 dist-test-slave-mp03 krb5kdc[13802](info): commencing operation
krb5kdc: starting...
W20260927 22:11:57.571036 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:11:57 dist-test-slave-mp03 krb5kdc[13802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:11:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:11:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:11:57.802991 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43537
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43537,127.0.97.61:36111,127.0.97.60:33197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:11:58.137511 13815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:11:58.137807 13815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:11:58.137964 13815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:11:58.145206 13815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:11:58.145318 13815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:11:58.145406 13815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:11:58.145494 13815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:11:58.145576 13815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:11:58.155671 13815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43537,127.0.97.61:36111,127.0.97.60:33197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:11:58.157155 13815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:11:58.158555 13815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:11:58.170833 13821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:11:58.171010 13823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:11:58.171301 13820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:11:58.172127 13815 server_base.cc:1034] running on GCE node
I20260927 22:11:58.172998 13815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:11:58.174566 13815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:11:58.175835 13815 hybrid_clock.cc:648] HybridClock initialized: now 1790547118175772 us; error 56 us; skew 500 ppm
I20260927 22:11:58.176268 13815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:11:58 dist-test-slave-mp03 krb5kdc[13802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:11:58.179548 13815 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:11:58.181011 13815 webserver.cc:460] Webserver started at http://127.0.97.62:40715/ using document root <none> and password file <none>
I20260927 22:11:58.181655 13815 fs_manager.cc:359] Metadata directory not provided
I20260927 22:11:58.181753 13815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:11:58.181994 13815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:11:58.185078 13815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "06f3599e94984a84bc1b34dd8154ced0"
format_stamp: "Formatted at 2026-09-27 22:11:58 on dist-test-slave-mp03"
server_key: "cdea788223cc3d57adb6566af56c2d24"
server_key_iv: "a1c09c22e6dbef5cd7391111dd9fe704"
server_key_version: "encryptionkey@0"
I20260927 22:11:58.185814 13815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "06f3599e94984a84bc1b34dd8154ced0"
format_stamp: "Formatted at 2026-09-27 22:11:58 on dist-test-slave-mp03"
server_key: "cdea788223cc3d57adb6566af56c2d24"
server_key_iv: "a1c09c22e6dbef5cd7391111dd9fe704"
server_key_version: "encryptionkey@0"
I20260927 22:11:58.191030 13815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:11:58.194880 13830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:11:58.196375 13815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:11:58.196564 13815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "06f3599e94984a84bc1b34dd8154ced0"
format_stamp: "Formatted at 2026-09-27 22:11:58 on dist-test-slave-mp03"
server_key: "cdea788223cc3d57adb6566af56c2d24"
server_key_iv: "a1c09c22e6dbef5cd7391111dd9fe704"
server_key_version: "encryptionkey@0"
I20260927 22:11:58.196791 13815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:11:58.223053 13815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:11:58.227396 13815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:11:58.227716 13815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:11:58.247030 13815 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43537
I20260927 22:11:58.247056 13881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43537 every 8 connection(s)
I20260927 22:11:58.248492 13815 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:11:58.252293 13882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:11:58.256332 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 13815
I20260927 22:11:58.256560 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:11:58.257373 388 external_mini_cluster.cc:1428] Setting key e7c052a809e6177d879c7c40df46070e
I20260927 22:11:58.260762 13882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:11:58 dist-test-slave-mp03 krb5kdc[13802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260927 22:11:58.288305 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.265030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42000 (local address 127.0.97.62:43537)
0927 22:11:58.267241 (+ 2211us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:58.267251 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:58.267303 (+ 52us) server_negotiation.cc:407] Connection header received
0927 22:11:58.268611 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:58.268621 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:58.268926 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:58.269154 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:58.270474 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:58.272526 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:58.274233 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:58.275318 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:58.275534 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:58.279596 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:58.279612 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:58.279633 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:58.279671 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:58.283161 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:58.284098 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:58.284108 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:58.284112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:58.284280 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:58.284747 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:58.284752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:58.284756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:58.285327 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:58.285567 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:58.286202 (+ 635us) server_negotiation.cc:299] Negotiation successful
0927 22:11:58.286657 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":124,"server-negotiator.queue_time_us":1877,"spinlock_wait_cycles":82176,"thread_start_us":1631,"threads_started":1}
I20260927 22:11:58.288355 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.265030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43537 (local address 127.0.0.1:42000)
0927 22:11:58.266049 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:58.266147 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:58.268216 (+ 2069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:58.269206 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:58.269211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:58.269872 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:11:58.270244 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:58.270263 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:58.272724 (+ 2461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:58.272729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:58.274056 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:58.274072 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:58.275605 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:58.275613 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:58.275839 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:58.276665 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:11:58.276692 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:11:58.279320 (+ 2628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:11:58.283307 (+ 3987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:58.283318 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:58.283328 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:58.283960 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:58.284419 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:58.284424 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:58.284428 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:58.284610 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:58.285503 (+ 893us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:11:58.285512 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:11:58.285770 (+ 258us) client_negotiation.cc:769] Sending connection context
0927 22:11:58.286063 (+ 293us) client_negotiation.cc:240] Negotiation successful
0927 22:11:58.286768 (+ 705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"mutex_wait_us":251,"thread_start_us":413,"threads_started":1}
I20260927 22:11:58.291597 13882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:11:58.292835 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.292382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36111 (local address 127.0.0.1:52098)
0927 22:11:58.292607 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:58.292717 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260927 22:11:58.293135 13833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36111: connect: Connection refused (error 111)
I20260927 22:11:58.294899 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.294504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36111 (local address 127.0.0.1:52102)
0927 22:11:58.294690 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:58.294763 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 22:11:58.295141 13882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36111: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36111: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260927 22:11:58.315145 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36111
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43537,127.0.97.61:36111,127.0.97.60:33197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:11:58.344476 13882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } attempt: 1
I20260927 22:11:58.345789 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.345243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36111 (local address 127.0.0.1:50422)
0927 22:11:58.345483 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:58.345610 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 22:11:58.347018 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.346571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36111 (local address 127.0.0.1:50436)
0927 22:11:58.346768 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:58.346855 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260927 22:11:58.347270 13882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36111: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36111: connect: Connection refused (error 111)
I20260927 22:11:58.415580 13882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } attempt: 2
I20260927 22:11:58.416972 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.416284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36111 (local address 127.0.0.1:50450)
0927 22:11:58.416629 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:58.416769 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260927 22:11:58.418133 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.417644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36111 (local address 127.0.0.1:50454)
0927 22:11:58.417851 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:58.417962 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260927 22:11:58.418341 13882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36111: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36111: connect: Connection refused (error 111)
I20260927 22:11:58.509642 13882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } attempt: 3
I20260927 22:11:58.511045 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.510363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36111 (local address 127.0.0.1:50464)
0927 22:11:58.510682 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:58.510820 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 22:11:58.512372 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.511818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36111 (local address 127.0.0.1:50476)
0927 22:11:58.512036 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:58.512149 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260927 22:11:58.512583 13882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36111: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36111: connect: Connection refused (error 111)
I20260927 22:11:58.655844 13882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } attempt: 4
W20260927 22:11:58.656003 13889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:11:58.656416 13889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:11:58.656584 13889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 22:11:58.657279 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.656620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36111 (local address 127.0.0.1:50478)
0927 22:11:58.657010 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:58.657118 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260927 22:11:58.658353 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.657955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36111 (local address 127.0.0.1:50480)
0927 22:11:58.658105 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:58.658209 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 22:11:58.658568 13882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36111: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36111: connect: Connection refused (error 111)
W20260927 22:11:58.664682 13889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:11:58.664812 13889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:11:58.664896 13889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:11:58.665004 13889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:11:58.665071 13889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:11:58.675449 13889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43537,127.0.97.61:36111,127.0.97.60:33197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:11:58.677111 13889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:11:58.678535 13889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:11:58.691273 13897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:11:58.691339 13895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:11:58.691278 13894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:11:58.698015 13889 server_base.cc:1034] running on GCE node
I20260927 22:11:58.698952 13889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:11:58.700601 13889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:11:58.701853 13889 hybrid_clock.cc:648] HybridClock initialized: now 1790547118701771 us; error 81 us; skew 500 ppm
I20260927 22:11:58.702224 13889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:11:58 dist-test-slave-mp03 krb5kdc[13802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:11:58.705389 13889 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260927 22:11:58.707027 13889 webserver.cc:460] Webserver started at http://127.0.97.61:34497/ using document root <none> and password file <none>
I20260927 22:11:58.707693 13889 fs_manager.cc:359] Metadata directory not provided
I20260927 22:11:58.707790 13889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:11:58.708029 13889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:11:58.711042 13889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-1/data/instance:
uuid: "f4c3c3852bd04b8ca8fed047c48d23f9"
format_stamp: "Formatted at 2026-09-27 22:11:58 on dist-test-slave-mp03"
server_key: "d1768a189d960661458504aa5acf9ed1"
server_key_iv: "ad611b3dfaa1e71d805f9b4256707cb8"
server_key_version: "encryptionkey@0"
I20260927 22:11:58.711829 13889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-1/wal/instance:
uuid: "f4c3c3852bd04b8ca8fed047c48d23f9"
format_stamp: "Formatted at 2026-09-27 22:11:58 on dist-test-slave-mp03"
server_key: "d1768a189d960661458504aa5acf9ed1"
server_key_iv: "ad611b3dfaa1e71d805f9b4256707cb8"
server_key_version: "encryptionkey@0"
I20260927 22:11:58.717084 13889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 22:11:58.720444 13904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:11:58.721807 13889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:11:58.721935 13889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-1/wal
uuid: "f4c3c3852bd04b8ca8fed047c48d23f9"
format_stamp: "Formatted at 2026-09-27 22:11:58 on dist-test-slave-mp03"
server_key: "d1768a189d960661458504aa5acf9ed1"
server_key_iv: "ad611b3dfaa1e71d805f9b4256707cb8"
server_key_version: "encryptionkey@0"
I20260927 22:11:58.722093 13889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:11:58.748775 13889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:11:58.753206 13889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:11:58.753459 13889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:11:58.772828 13889 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:36111
I20260927 22:11:58.772861 13955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:36111 every 8 connection(s)
I20260927 22:11:58.774251 13889 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-1/data/info.pb
I20260927 22:11:58.778083 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 13889
I20260927 22:11:58.778314 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-1/wal/instance
I20260927 22:11:58.778759 13956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:11:58.778887 388 external_mini_cluster.cc:1428] Setting key fb5ca032b7bc2c4b6faf2e8070e5b4fb
I20260927 22:11:58.785655 13956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 27 22:11:58 dist-test-slave-mp03 krb5kdc[13802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:11:58.808203 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.788459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58644 (local address 127.0.97.62:43537)
0927 22:11:58.788696 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:58.788702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:58.790051 (+ 1349us) server_negotiation.cc:407] Connection header received
0927 22:11:58.792107 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:58.792116 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:58.792233 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:58.792364 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:58.793954 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:58.795240 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:58.796823 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:58.797575 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:58.797706 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:58.802081 (+ 4375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:58.802086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:58.802089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:58.802126 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:58.804968 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:58.805760 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:58.805766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:58.805770 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:58.805900 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:58.806334 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:58.806340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:58.806343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:58.806727 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:58.806900 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:58.807619 (+ 719us) server_negotiation.cc:299] Negotiation successful
0927 22:11:58.807853 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 22:11:58.809336 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.788970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43537 (local address 127.0.0.1:58644)
0927 22:11:58.789840 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:58.789895 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:58.791779 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:58.792574 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:58.792583 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:58.793159 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:11:58.793756 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:58.793774 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:58.795409 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:58.795413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:58.796669 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:58.796681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:58.797802 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:58.797807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:58.798065 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:58.798804 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:11:58.798833 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:11:58.801821 (+ 2988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:11:58.805134 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:58.805144 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:58.805167 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:58.805611 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:58.806021 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:58.806026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:58.806030 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:58.806217 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:58.806862 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:11:58.806870 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:11:58.807248 (+ 378us) client_negotiation.cc:769] Sending connection context
0927 22:11:58.807534 (+ 286us) client_negotiation.cc:240] Negotiation successful
0927 22:11:58.808099 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":399,"threads_started":1}
I20260927 22:11:58.811261 13956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:11:58 dist-test-slave-mp03 krb5kdc[13802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:11:58.833079 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.812151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36111 (local address 127.0.0.1:50490)
0927 22:11:58.812348 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:58.812375 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:58.812509 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:58.815570 (+ 3061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:58.815577 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:58.816088 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:11:58.816300 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:58.816313 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:58.817679 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:58.817687 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:58.819354 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:58.819373 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:58.820285 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:58.820291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:58.820491 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:58.821250 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:11:58.821278 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:11:58.826369 (+ 5091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:11:58.830344 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:58.830349 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:58.830351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:58.830971 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:58.831441 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:58.831445 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:58.831447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:58.831600 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:58.832275 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:11:58.832282 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:11:58.832414 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 22:11:58.832538 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 22:11:58.832788 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 22:11:58.833381 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.812163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50490 (local address 127.0.97.61:36111)
0927 22:11:58.815124 (+ 2961us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:58.815130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:58.815157 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:11:58.815200 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:58.815206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:58.815392 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:58.815508 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:58.816475 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:58.817506 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:58.819504 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:58.820052 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:58.820198 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:58.826716 (+ 6518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:58.826732 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:58.826754 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:58.826824 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:58.830172 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:58.831147 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:58.831154 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:58.831156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:58.831311 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:58.831714 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:58.831717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:58.831719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:58.832163 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:58.832339 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:58.832757 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 22:11:58.832959 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2798,"thread_start_us":2631,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260927 22:11:58.834733 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33197
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43537,127.0.97.61:36111,127.0.97.60:33197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:11:58.836062 13956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:11:58.837400 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.836792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33197 (local address 127.0.0.1:60992)
0927 22:11:58.837023 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:58.837205 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260927 22:11:58.837713 13908 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
I20260927 22:11:58.839392 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.838963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33197 (local address 127.0.0.1:60994)
0927 22:11:58.839137 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:58.839242 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 22:11:58.839610 13956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33197: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
I20260927 22:11:58.888947 13956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } attempt: 1
I20260927 22:11:58.890187 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.889664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33197 (local address 127.0.0.1:32776)
0927 22:11:58.889884 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:58.889988 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 22:11:58.891284 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.890921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33197 (local address 127.0.0.1:32790)
0927 22:11:58.891105 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:58.891173 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 22:11:58.891494 13956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33197: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
I20260927 22:11:58.957912 13882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } attempt: 5
I20260927 22:11:58.959733 13956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } attempt: 2
I20260927 22:11:58.960745 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.960274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33197 (local address 127.0.0.1:32804)
0927 22:11:58.960471 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:58.960579 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 22:11:58.961735 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.961409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33197 (local address 127.0.0.1:32818)
0927 22:11:58.961550 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:58.961618 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 22:11:58.963079 13956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33197: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
Sep 27 22:11:58 dist-test-slave-mp03 krb5kdc[13802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:11:58.972839 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.958743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36111 (local address 127.0.0.1:50496)
0927 22:11:58.958984 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:58.959006 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:58.959144 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:58.959704 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:58.959712 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:58.960200 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:11:58.960435 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:58.960448 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:58.961865 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:58.961871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:58.962821 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:58.962841 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:58.963834 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:58.963838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:58.963978 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:58.964607 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:11:58.964624 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:11:58.966678 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:11:58.970394 (+ 3716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:58.970398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:58.970401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:58.970846 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:58.971252 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:58.971255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:58.971258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:58.971382 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:58.972086 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:11:58.972094 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:11:58.972267 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 22:11:58.972385 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 22:11:58.972591 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 22:11:58.973278 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.958880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50496 (local address 127.0.97.61:36111)
0927 22:11:58.959104 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:58.959110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:58.959133 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:11:58.959326 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:58.959331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:58.959437 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:58.959551 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:58.960595 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:58.961699 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:58.962965 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:58.963667 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:58.963818 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:58.966860 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:58.966867 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:58.966871 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:58.966909 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:58.970248 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:58.970978 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:58.970983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:58.970987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:58.971130 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:58.971499 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:58.971503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:58.971506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:58.971855 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:58.972023 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:58.972561 (+ 538us) server_negotiation.cc:299] Negotiation successful
0927 22:11:58.972781 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:11:58.974321 13882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:11:58.975533 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.975136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33197 (local address 127.0.0.1:32822)
0927 22:11:58.975315 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:58.975401 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 22:11:58.976490 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:58.976177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33197 (local address 127.0.0.1:32824)
0927 22:11:58.976329 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:58.976401 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 22:11:58.976688 13882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33197: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
I20260927 22:11:59.027911 13882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } attempt: 1
I20260927 22:11:59.029036 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.028571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33197 (local address 127.0.0.1:32840)
0927 22:11:59.028770 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:59.028875 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 22:11:59.030040 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.029686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33197 (local address 127.0.0.1:32852)
0927 22:11:59.029852 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:59.029933 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260927 22:11:59.030270 13882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33197: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
I20260927 22:11:59.054333 13956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } attempt: 3
I20260927 22:11:59.055521 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.055041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33197 (local address 127.0.0.1:32858)
0927 22:11:59.055269 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:59.055362 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 22:11:59.056581 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.056144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33197 (local address 127.0.0.1:32860)
0927 22:11:59.056401 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:59.056478 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260927 22:11:59.056787 13956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33197: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
I20260927 22:11:59.098489 13882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } attempt: 2
I20260927 22:11:59.099756 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.099252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33197 (local address 127.0.0.1:32864)
0927 22:11:59.099483 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:59.099577 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 22:11:59.100747 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.100399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33197 (local address 127.0.0.1:32866)
0927 22:11:59.100548 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:59.100634 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 22:11:59.101006 13882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33197: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
W20260927 22:11:59.174508 13963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:11:59.174942 13963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:11:59.175104 13963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:11:59.182407 13963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:11:59.182576 13963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:11:59.182689 13963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:11:59.182796 13963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:11:59.182859 13963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:11:59.192981 13963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43537,127.0.97.61:36111,127.0.97.60:33197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:11:59.194425 13963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:11:59.195814 13963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 22:11:59.199991 13956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } attempt: 4
I20260927 22:11:59.201299 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.200713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33197 (local address 127.0.0.1:32874)
0927 22:11:59.200981 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:59.201090 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 22:11:59.202493 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.202057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33197 (local address 127.0.0.1:32888)
0927 22:11:59.202286 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:59.202375 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260927 22:11:59.202813 13956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33197: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
I20260927 22:11:59.207198 13882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } attempt: 3
I20260927 22:11:59.208393 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.207794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33197 (local address 127.0.0.1:32902)
0927 22:11:59.208021 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:59.208213 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260927 22:11:59.208990 13968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:11:59.209200 13971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:11:59.209621 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.209134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33197 (local address 127.0.0.1:32908)
0927 22:11:59.209395 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:59.209481 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260927 22:11:59.209589 13969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:11:59.209834 13963 server_base.cc:1034] running on GCE node
W20260927 22:11:59.209895 13882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33197: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33197: connect: Connection refused (error 111)
I20260927 22:11:59.210690 13963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:11:59.212123 13963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:11:59.213328 13963 hybrid_clock.cc:648] HybridClock initialized: now 1790547119213279 us; error 42 us; skew 500 ppm
I20260927 22:11:59.213685 13963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:11:59 dist-test-slave-mp03 krb5kdc[13802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:11:59.216742 13963 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260927 22:11:59.218087 13963 webserver.cc:460] Webserver started at http://127.0.97.60:36999/ using document root <none> and password file <none>
I20260927 22:11:59.218782 13963 fs_manager.cc:359] Metadata directory not provided
I20260927 22:11:59.218881 13963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:11:59.219135 13963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:11:59.221985 13963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-2/data/instance:
uuid: "bc33ade617ea4b1cbb78715696ea5f43"
format_stamp: "Formatted at 2026-09-27 22:11:59 on dist-test-slave-mp03"
server_key: "8a1afb6c22c57c08ad7047127e32b5f3"
server_key_iv: "bd8de13276bb266767ae3b2a191f325a"
server_key_version: "encryptionkey@0"
I20260927 22:11:59.222807 13963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-2/wal/instance:
uuid: "bc33ade617ea4b1cbb78715696ea5f43"
format_stamp: "Formatted at 2026-09-27 22:11:59 on dist-test-slave-mp03"
server_key: "8a1afb6c22c57c08ad7047127e32b5f3"
server_key_iv: "bd8de13276bb266767ae3b2a191f325a"
server_key_version: "encryptionkey@0"
I20260927 22:11:59.228368 13963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:11:59.231966 13978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:11:59.233489 13963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:11:59.233662 13963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-2/wal
uuid: "bc33ade617ea4b1cbb78715696ea5f43"
format_stamp: "Formatted at 2026-09-27 22:11:59 on dist-test-slave-mp03"
server_key: "8a1afb6c22c57c08ad7047127e32b5f3"
server_key_iv: "bd8de13276bb266767ae3b2a191f325a"
server_key_version: "encryptionkey@0"
I20260927 22:11:59.233848 13963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:11:59.261969 13963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:11:59.267264 13963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:11:59.267655 13963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:11:59.286952 13963 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:33197
I20260927 22:11:59.286983 14029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:33197 every 8 connection(s)
I20260927 22:11:59.288435 13963 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-2/data/info.pb
I20260927 22:11:59.292244 14030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:11:59.298408 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 13963
I20260927 22:11:59.298684 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546912983837-388-0/minicluster-data/master-2/wal/instance
I20260927 22:11:59.299140 388 external_mini_cluster.cc:1428] Setting key a030d14608ef5622875a6d3854189fd9
I20260927 22:11:59.298875 14030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:11:59 dist-test-slave-mp03 krb5kdc[13802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 27 22:11:59 dist-test-slave-mp03 krb5kdc[13802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 22:11:59.322119 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.301694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58646 (local address 127.0.97.62:43537)
0927 22:11:59.301974 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:59.301982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:59.302856 (+ 874us) server_negotiation.cc:407] Connection header received
0927 22:11:59.304072 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:59.304078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:59.304163 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:59.304281 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:59.306024 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.307308 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.309422 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.310136 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.310283 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:59.314887 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:59.314896 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:59.314903 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:59.314951 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:59.318161 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:59.319344 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:59.319348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:59.319351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:59.319485 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:59.319949 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:59.319955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:59.319959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:59.320350 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:59.320491 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:59.321485 (+ 994us) server_negotiation.cc:299] Negotiation successful
0927 22:11:59.321733 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260927 22:11:59.323433 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.301840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43537 (local address 127.0.0.1:58646)
0927 22:11:59.302702 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:59.302741 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:59.303824 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:59.304382 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:59.304388 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:59.305234 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:11:59.305788 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:59.305816 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.307482 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.307486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:59.309260 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:59.309276 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.310363 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.310368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:59.310668 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:59.311552 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:11:59.311584 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:11:59.314588 (+ 3004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:11:59.318337 (+ 3749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:59.318348 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:59.318361 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:59.319209 (+ 848us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:59.319614 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:59.319620 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:59.319625 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:59.319814 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:59.320494 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:11:59.320502 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:11:59.321016 (+ 514us) client_negotiation.cc:769] Sending connection context
0927 22:11:59.321352 (+ 336us) client_negotiation.cc:240] Negotiation successful
0927 22:11:59.321928 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":65,"thread_start_us":427,"threads_started":1}
I20260927 22:11:59.323464 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.300982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32914 (local address 127.0.97.60:33197)
0927 22:11:59.301943 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:59.301953 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:59.301989 (+ 36us) server_negotiation.cc:407] Connection header received
0927 22:11:59.303806 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:59.303827 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:59.304132 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:59.304342 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:59.306172 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.307886 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.309644 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.310569 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.310745 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:59.315615 (+ 4870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:59.315624 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:59.315637 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:59.315668 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:59.319483 (+ 3815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:59.320333 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:59.320340 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:59.320342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:59.320526 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:59.320918 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:59.320922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:59.320925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:59.321388 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:59.321690 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:59.321998 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 22:11:59.322222 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":779,"thread_start_us":552,"threads_started":1}
I20260927 22:11:59.325115 14030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:11:59 dist-test-slave-mp03 krb5kdc[13802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 27 22:11:59 dist-test-slave-mp03 krb5kdc[13802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:11:59.341167 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.326070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50512 (local address 127.0.97.61:36111)
0927 22:11:59.326347 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:59.326354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:59.326373 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:11:59.326433 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:59.326436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:59.326493 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:59.326604 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:59.327444 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.328577 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.330224 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.330899 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.331026 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:59.334697 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:59.334701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:59.334704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:59.334737 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:59.338402 (+ 3665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:59.339195 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:59.339202 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:59.339206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:59.339346 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:59.339803 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:59.339809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:59.339812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:59.340134 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:59.340302 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:59.340725 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 22:11:59.340890 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 22:11:59.341256 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.325880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36111 (local address 127.0.0.1:50512)
0927 22:11:59.326064 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:59.326083 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:59.326180 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:59.326667 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:59.326671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:59.327118 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:11:59.327305 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:59.327316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.328730 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.328734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:59.330079 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:59.330096 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.331050 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.331055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:59.331212 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:59.331846 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:11:59.331863 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:11:59.334456 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:11:59.338556 (+ 4100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:59.338560 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:59.338563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:59.339067 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:59.339464 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:59.339470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:59.339473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:59.339685 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:59.340259 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:11:59.340264 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:11:59.340417 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 22:11:59.340593 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 22:11:59.340875 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 22:11:59.342844 14030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:11:59.343653 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.326834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58658 (local address 127.0.97.62:43537)
0927 22:11:59.327180 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:59.327185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:59.327204 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:11:59.327296 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:59.327301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:59.327373 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:59.327462 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:59.328941 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.330019 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.331721 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.332116 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.332209 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:59.337031 (+ 4822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:59.337102 (+ 71us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:59.337109 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:59.337187 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:59.340427 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:59.341272 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:59.341278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:59.341281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:59.341453 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:59.341907 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:59.341912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:59.341918 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:59.342352 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:59.342734 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:59.343055 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 22:11:59.343286 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 22:11:59 dist-test-slave-mp03 krb5kdc[13802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 22:11:59 dist-test-slave-mp03 krb5kdc[13802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:11:59.358027 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.343866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33197 (local address 127.0.0.1:32918)
0927 22:11:59.344124 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:59.344139 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:59.344282 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:59.344621 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:59.344626 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:59.345008 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:11:59.345143 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:59.345183 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.346491 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.346508 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:59.347822 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:59.347835 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.348852 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.348858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:59.349051 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:59.349680 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:11:59.349705 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:11:59.352199 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:11:59.355413 (+ 3214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:59.355420 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:59.355424 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:59.355781 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:59.356183 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:59.356191 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:59.356197 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:59.356325 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:59.356964 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:11:59.356969 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:11:59.357114 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 22:11:59.357280 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 22:11:59.357618 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260927 22:11:59.358024 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.343723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32918 (local address 127.0.97.60:33197)
0927 22:11:59.343939 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:59.343952 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:59.344220 (+ 268us) server_negotiation.cc:407] Connection header received
0927 22:11:59.344416 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:59.344423 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:59.344491 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:59.344609 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:59.345343 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.346337 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.347970 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.348678 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.348803 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:59.352402 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:59.352408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:59.352411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:59.352438 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:59.355281 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:59.355909 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:59.355916 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:59.355920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:59.356052 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:59.356433 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:59.356441 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:59.356444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:59.356842 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:59.357003 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:59.357358 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 22:11:59.357726 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":173,"server-negotiator.queue_time_us":99}
I20260927 22:11:59.361986 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.345392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50526 (local address 127.0.97.61:36111)
0927 22:11:59.345628 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:59.345635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:59.345654 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:11:59.345761 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:59.345767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:59.345837 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:59.345936 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:59.347531 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.348589 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.350580 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.351267 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.351427 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:59.356017 (+ 4590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:59.356024 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:59.356034 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:59.356068 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:59.359176 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:59.360171 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:59.360175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:59.360177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:59.360314 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:59.360737 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:59.360740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:59.360742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:59.360986 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:59.361198 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:59.361546 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 22:11:59.361722 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 22:11:59.364439 14030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43: Bootstrap starting.
I20260927 22:11:59.367165 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:11:59.368077 14030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43: Neither blocks nor log segments found. Creating new log.
I20260927 22:11:59.369089 14030 log.cc:824] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43: Log is configured to *not* fsync() on all Append() calls
I20260927 22:11:59.372231 14030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43: No bootstrap required, opened a new log
I20260927 22:11:59.384169 14030 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f3599e94984a84bc1b34dd8154ced0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } } peers { permanent_uuid: "f4c3c3852bd04b8ca8fed047c48d23f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } } peers { permanent_uuid: "bc33ade617ea4b1cbb78715696ea5f43" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } }
I20260927 22:11:59.384728 14030 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:11:59.384914 14030 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc33ade617ea4b1cbb78715696ea5f43, State: Initialized, Role: FOLLOWER
I20260927 22:11:59.385690 14030 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f3599e94984a84bc1b34dd8154ced0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } } peers { permanent_uuid: "f4c3c3852bd04b8ca8fed047c48d23f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } } peers { permanent_uuid: "bc33ade617ea4b1cbb78715696ea5f43" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } }
I20260927 22:11:59.387112 13882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } attempt: 4
I20260927 22:11:59.396966 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.372246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32932 (local address 127.0.97.60:33197)
0927 22:11:59.372419 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:59.372426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:59.377417 (+ 4991us) server_negotiation.cc:407] Connection header received
0927 22:11:59.377985 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:59.377995 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:59.378087 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:59.378177 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:59.380786 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.382001 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.384260 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.384982 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.385078 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:59.389001 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:59.389009 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:59.389016 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:59.389056 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:59.393341 (+ 4285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:59.394257 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:59.394263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:59.394267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:59.394439 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:59.394942 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:59.394949 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:59.394953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:59.395473 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:59.395732 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:59.396321 (+ 589us) server_negotiation.cc:299] Negotiation successful
0927 22:11:59.396559 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 22:11:59.399076 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.372163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50534 (local address 127.0.97.61:36111)
0927 22:11:59.372336 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:59.372344 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:59.379469 (+ 7125us) server_negotiation.cc:407] Connection header received
0927 22:11:59.379928 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:59.379935 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:59.380021 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:59.380097 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:59.382557 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.383704 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.386168 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.386920 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.387030 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:59.391240 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:59.391253 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:59.391261 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:59.391313 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:59.395035 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:59.395899 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:59.395904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:59.395908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:59.396065 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:59.396516 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:59.396521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:59.396525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:59.396922 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:59.397169 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:59.398188 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0927 22:11:59.398727 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:11:59.401623 14043 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f3599e94984a84bc1b34dd8154ced0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } } peers { permanent_uuid: "f4c3c3852bd04b8ca8fed047c48d23f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } } peers { permanent_uuid: "bc33ade617ea4b1cbb78715696ea5f43" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } } }
I20260927 22:11:59.402299 14043 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:11:59.403190 14030 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:11:59.406854 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.372540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58672 (local address 127.0.97.62:43537)
0927 22:11:59.372715 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:59.372723 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:59.376126 (+ 3403us) server_negotiation.cc:407] Connection header received
0927 22:11:59.376333 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:59.376339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:59.376424 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:59.376528 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:59.379103 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.380230 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.382482 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.383135 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.383228 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:59.397272 (+ 14044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:59.397283 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:59.397291 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:59.397362 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:59.403781 (+ 6419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:59.404715 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:59.404721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:59.404726 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:59.404888 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:59.405306 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:59.405310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:59.405314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:59.405618 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:59.405909 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:59.406224 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 22:11:59.406458 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 27 22:11:59 dist-test-slave-mp03 krb5kdc[13802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 22:11:59.419241 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.387850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33197 (local address 127.0.0.1:32944)
0927 22:11:59.388308 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:59.388332 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:59.388458 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:59.398320 (+ 9862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:59.398329 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:59.398957 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:11:59.399274 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:59.399291 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.403938 (+ 4647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.403946 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:59.405810 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:59.405837 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.407689 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.407698 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:59.407994 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:59.408713 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:11:59.408744 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:11:59.411988 (+ 3244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:11:59.416110 (+ 4122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:59.416117 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:59.416121 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:59.416664 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:59.417150 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:59.417157 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:59.417164 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:59.417378 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:59.418092 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:11:59.418097 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:11:59.418271 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 22:11:59.418434 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 22:11:59.418819 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291}
I20260927 22:11:59.420269 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.388705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32944 (local address 127.0.97.60:33197)
0927 22:11:59.397618 (+ 8913us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:59.397623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:59.397639 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:11:59.397684 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:59.397689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:59.397779 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:59.397873 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:59.400736 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.402017 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.406046 (+ 4029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.407257 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.407411 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:59.412228 (+ 4817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:59.412234 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:59.412237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:59.412271 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:59.415915 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:59.416815 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:59.416821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:59.416825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:59.416994 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:59.417518 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:59.417523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:59.417526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:59.417917 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:59.418054 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:59.419559 (+ 1505us) server_negotiation.cc:299] Negotiation successful
0927 22:11:59.419820 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8788,"thread_start_us":14006,"threads_started":1}
W20260927 22:11:59.421363 14055 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:11:59.421553 14055 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:11:59.427505 13882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06f3599e94984a84bc1b34dd8154ced0: Bootstrap starting.
I20260927 22:11:59.418130 13963 master_runner.cc:186] Error getting master registration for 127.0.97.62:43537: OK, instance_id { permanent_uuid: "06f3599e94984a84bc1b34dd8154ced0" instance_seqno: 1790547118241452 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:11:59.430965 13963 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:11:59.431921 13882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06f3599e94984a84bc1b34dd8154ced0: Neither blocks nor log segments found. Creating new log.
I20260927 22:11:59.433388 13882 log.cc:824] T 00000000000000000000000000000000 P 06f3599e94984a84bc1b34dd8154ced0: Log is configured to *not* fsync() on all Append() calls
I20260927 22:11:59.444208 13882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06f3599e94984a84bc1b34dd8154ced0: No bootstrap required, opened a new log
I20260927 22:11:59.455406 13882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06f3599e94984a84bc1b34dd8154ced0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f3599e94984a84bc1b34dd8154ced0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } } peers { permanent_uuid: "f4c3c3852bd04b8ca8fed047c48d23f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } } peers { permanent_uuid: "bc33ade617ea4b1cbb78715696ea5f43" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } }
I20260927 22:11:59.455808 13882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06f3599e94984a84bc1b34dd8154ced0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:11:59.455937 13882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06f3599e94984a84bc1b34dd8154ced0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f3599e94984a84bc1b34dd8154ced0, State: Initialized, Role: FOLLOWER
I20260927 22:11:59.456653 13882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06f3599e94984a84bc1b34dd8154ced0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f3599e94984a84bc1b34dd8154ced0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } } peers { permanent_uuid: "f4c3c3852bd04b8ca8fed047c48d23f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } } peers { permanent_uuid: "bc33ade617ea4b1cbb78715696ea5f43" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } }
I20260927 22:11:59.457625 14058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06f3599e94984a84bc1b34dd8154ced0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f3599e94984a84bc1b34dd8154ced0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } } peers { permanent_uuid: "f4c3c3852bd04b8ca8fed047c48d23f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } } peers { permanent_uuid: "bc33ade617ea4b1cbb78715696ea5f43" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } } }
I20260927 22:11:59.457984 14058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06f3599e94984a84bc1b34dd8154ced0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:11:59.459087 13882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06f3599e94984a84bc1b34dd8154ced0 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 22:11:59.471961 14069 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06f3599e94984a84bc1b34dd8154ced0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:11:59.472179 14069 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 22:11:59.480836 13836 tablet.cc:2369] T 00000000000000000000000000000000 P 06f3599e94984a84bc1b34dd8154ced0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:11:59.470310 13815 master_runner.cc:186] Error getting master registration for 127.0.97.61:36111: OK, instance_id { permanent_uuid: "f4c3c3852bd04b8ca8fed047c48d23f9" instance_seqno: 1790547118767117 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:11:59.483354 13815 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:11:59.502081 13956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } attempt: 5
Sep 27 22:11:59 dist-test-slave-mp03 krb5kdc[13802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260927 22:11:59.520047 13984 tablet.cc:2369] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:11:59.523005 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.502796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33197 (local address 127.0.0.1:32960)
0927 22:11:59.503208 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 22:11:59.503232 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:11:59.503354 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:11:59.504337 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:11:59.504343 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:11:59.504953 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:11:59.505182 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:59.505195 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.506719 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.506725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:59.508345 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:11:59.508362 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.509384 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.509392 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:11:59.509632 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:59.510384 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:11:59.510412 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:11:59.513587 (+ 3175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:11:59.518489 (+ 4902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:59.518495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:59.518499 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:59.519157 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:59.519696 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:11:59.519704 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:11:59.519710 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:11:59.521191 (+ 1481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:11:59.521913 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:11:59.521918 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:11:59.522098 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 22:11:59.522263 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 22:11:59.522568 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271}
I20260927 22:11:59.523524 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:11:59.503629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32960 (local address 127.0.97.60:33197)
0927 22:11:59.503836 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:11:59.503843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:11:59.503863 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:11:59.503923 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:11:59.503930 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:11:59.504015 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:11:59.504123 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:11:59.505381 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.506540 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.508502 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:11:59.509213 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:11:59.509340 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:11:59.513846 (+ 4506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:11:59.513854 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:11:59.513860 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:11:59.513905 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:11:59.518264 (+ 4359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:59.519324 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:59.519330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:59.519334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:59.519524 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:11:59.521359 (+ 1835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:11:59.521367 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:11:59.521371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:11:59.521762 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:11:59.521936 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 22:11:59.522397 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 22:11:59.522726 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:11:59.530131 13956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c3c3852bd04b8ca8fed047c48d23f9: Bootstrap starting.
I20260927 22:11:59.534720 13956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4c3c3852bd04b8ca8fed047c48d23f9: Neither blocks nor log segments found. Creating new log.
I20260927 22:11:59.536188 13956 log.cc:824] T 00000000000000000000000000000000 P f4c3c3852bd04b8ca8fed047c48d23f9: Log is configured to *not* fsync() on all Append() calls
I20260927 22:11:59.542143 13956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c3c3852bd04b8ca8fed047c48d23f9: No bootstrap required, opened a new log
I20260927 22:11:59.555018 13956 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4c3c3852bd04b8ca8fed047c48d23f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f3599e94984a84bc1b34dd8154ced0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } } peers { permanent_uuid: "f4c3c3852bd04b8ca8fed047c48d23f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } } peers { permanent_uuid: "bc33ade617ea4b1cbb78715696ea5f43" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } }
I20260927 22:11:59.555608 13956 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4c3c3852bd04b8ca8fed047c48d23f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:11:59.555758 13956 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4c3c3852bd04b8ca8fed047c48d23f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c3c3852bd04b8ca8fed047c48d23f9, State: Initialized, Role: FOLLOWER
I20260927 22:11:59.556528 13956 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4c3c3852bd04b8ca8fed047c48d23f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f3599e94984a84bc1b34dd8154ced0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } } peers { permanent_uuid: "f4c3c3852bd04b8ca8fed047c48d23f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } } peers { permanent_uuid: "bc33ade617ea4b1cbb78715696ea5f43" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } }
I20260927 22:11:59.558269 14072 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c3c3852bd04b8ca8fed047c48d23f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f3599e94984a84bc1b34dd8154ced0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } } peers { permanent_uuid: "f4c3c3852bd04b8ca8fed047c48d23f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } } peers { permanent_uuid: "bc33ade617ea4b1cbb78715696ea5f43" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } } }
I20260927 22:11:59.558735 14072 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c3c3852bd04b8ca8fed047c48d23f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:11:59.560020 13956 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4c3c3852bd04b8ca8fed047c48d23f9 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 22:11:59.578181 14083 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f4c3c3852bd04b8ca8fed047c48d23f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:11:59.578436 14083 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:11:59.579435 13889 master_runner.cc:263] No leader master found from master f4c3c3852bd04b8ca8fed047c48d23f9
I20260927 22:11:59.579582 13889 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:11:59.601567 14043 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:11:59.601933 14043 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f3599e94984a84bc1b34dd8154ced0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } } peers { permanent_uuid: "f4c3c3852bd04b8ca8fed047c48d23f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } } peers { permanent_uuid: "bc33ade617ea4b1cbb78715696ea5f43" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } }
I20260927 22:11:59.603864 14043 leader_election.cc:288] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06f3599e94984a84bc1b34dd8154ced0 (127.0.97.62:43537), f4c3c3852bd04b8ca8fed047c48d23f9 (127.0.97.61:36111)
I20260927 22:11:59.605036 13857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc33ade617ea4b1cbb78715696ea5f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f3599e94984a84bc1b34dd8154ced0" is_pre_election: true
I20260927 22:11:59.605659 13857 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 06f3599e94984a84bc1b34dd8154ced0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc33ade617ea4b1cbb78715696ea5f43 in term 0.
I20260927 22:11:59.606892 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc33ade617ea4b1cbb78715696ea5f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c3c3852bd04b8ca8fed047c48d23f9" is_pre_election: true
I20260927 22:11:59.607463 13931 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f4c3c3852bd04b8ca8fed047c48d23f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc33ade617ea4b1cbb78715696ea5f43 in term 0.
I20260927 22:11:59.608297 13979 leader_election.cc:302] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06f3599e94984a84bc1b34dd8154ced0, bc33ade617ea4b1cbb78715696ea5f43; no voters:
I20260927 22:11:59.609134 14043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:11:59.609278 14043 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:11:59.609369 14043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:11:59.612771 14043 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f3599e94984a84bc1b34dd8154ced0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } } peers { permanent_uuid: "f4c3c3852bd04b8ca8fed047c48d23f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } } peers { permanent_uuid: "bc33ade617ea4b1cbb78715696ea5f43" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } }
I20260927 22:11:59.613942 13857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc33ade617ea4b1cbb78715696ea5f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f3599e94984a84bc1b34dd8154ced0"
I20260927 22:11:59.614158 13857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06f3599e94984a84bc1b34dd8154ced0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:11:59.617543 13857 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 06f3599e94984a84bc1b34dd8154ced0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc33ade617ea4b1cbb78715696ea5f43 in term 1.
I20260927 22:11:59.619165 13979 leader_election.cc:302] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06f3599e94984a84bc1b34dd8154ced0, bc33ade617ea4b1cbb78715696ea5f43; no voters:
I20260927 22:11:59.622817 13931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc33ade617ea4b1cbb78715696ea5f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c3c3852bd04b8ca8fed047c48d23f9"
I20260927 22:11:59.623046 13931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4c3c3852bd04b8ca8fed047c48d23f9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:11:59.626621 13931 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f4c3c3852bd04b8ca8fed047c48d23f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc33ade617ea4b1cbb78715696ea5f43 in term 1.
I20260927 22:11:59.627367 14043 leader_election.cc:288] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [CANDIDATE]: Term 1 election: Requested vote from peers 06f3599e94984a84bc1b34dd8154ced0 (127.0.97.62:43537), f4c3c3852bd04b8ca8fed047c48d23f9 (127.0.97.61:36111)
I20260927 22:11:59.627763 14043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:11:59.628196 14043 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [term 1 LEADER]: Becoming Leader. State: Replica: bc33ade617ea4b1cbb78715696ea5f43, State: Running, Role: LEADER
I20260927 22:11:59.629686 14043 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f3599e94984a84bc1b34dd8154ced0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } } peers { permanent_uuid: "f4c3c3852bd04b8ca8fed047c48d23f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } } peers { permanent_uuid: "bc33ade617ea4b1cbb78715696ea5f43" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } }
I20260927 22:11:59.640838 14043 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc33ade617ea4b1cbb78715696ea5f43. Latest consensus state: current_term: 1 leader_uuid: "bc33ade617ea4b1cbb78715696ea5f43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f3599e94984a84bc1b34dd8154ced0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } } peers { permanent_uuid: "f4c3c3852bd04b8ca8fed047c48d23f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } } peers { permanent_uuid: "bc33ade617ea4b1cbb78715696ea5f43" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } } }
I20260927 22:11:59.641746 14043 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [sys.catalog]: This master's current role is: LEADER
I20260927 22:11:59.655323 14087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:11:59.657644 14087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:11:59.674158 13931 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f4c3c3852bd04b8ca8fed047c48d23f9 [term 1 FOLLOWER]: Refusing update from remote peer bc33ade617ea4b1cbb78715696ea5f43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:11:59.675514 14043 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c3c3852bd04b8ca8fed047c48d23f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:11:59.682996 13857 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 06f3599e94984a84bc1b34dd8154ced0 [term 1 FOLLOWER]: Refusing update from remote peer bc33ade617ea4b1cbb78715696ea5f43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:11:59.684106 14043 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f3599e94984a84bc1b34dd8154ced0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:11:59.704589 14058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06f3599e94984a84bc1b34dd8154ced0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc33ade617ea4b1cbb78715696ea5f43. Latest consensus state: current_term: 1 leader_uuid: "bc33ade617ea4b1cbb78715696ea5f43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f3599e94984a84bc1b34dd8154ced0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } } peers { permanent_uuid: "f4c3c3852bd04b8ca8fed047c48d23f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } } peers { permanent_uuid: "bc33ade617ea4b1cbb78715696ea5f43" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } } }
I20260927 22:11:59.704989 14058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06f3599e94984a84bc1b34dd8154ced0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:11:59.711802 14088 mvcc.cc:205] Tried to move back new op lower bound from 7334081002170261504 to 7334081002014470144. Current Snapshot: MvccSnapshot[applied={T|T < 7334081002170261504}]
I20260927 22:11:59.721942 14087 catalog_manager.cc:1347] Generated new cluster ID: f2b94c6499d5406982968c7f8a4b3680
I20260927 22:11:59.722085 14087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:11:59.722939 14085 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc33ade617ea4b1cbb78715696ea5f43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f3599e94984a84bc1b34dd8154ced0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } } peers { permanent_uuid: "f4c3c3852bd04b8ca8fed047c48d23f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } } peers { permanent_uuid: "bc33ade617ea4b1cbb78715696ea5f43" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } } }
I20260927 22:11:59.723512 14085 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [sys.catalog]: This master's current role is: LEADER
I20260927 22:11:59.733560 14072 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c3c3852bd04b8ca8fed047c48d23f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc33ade617ea4b1cbb78715696ea5f43. Latest consensus state: current_term: 1 leader_uuid: "bc33ade617ea4b1cbb78715696ea5f43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f3599e94984a84bc1b34dd8154ced0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } } peers { permanent_uuid: "f4c3c3852bd04b8ca8fed047c48d23f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } } peers { permanent_uuid: "bc33ade617ea4b1cbb78715696ea5f43" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } } }
I20260927 22:11:59.733932 14072 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c3c3852bd04b8ca8fed047c48d23f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:11:59.736887 14043 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc33ade617ea4b1cbb78715696ea5f43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f3599e94984a84bc1b34dd8154ced0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } } peers { permanent_uuid: "f4c3c3852bd04b8ca8fed047c48d23f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } } peers { permanent_uuid: "bc33ade617ea4b1cbb78715696ea5f43" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } } }
I20260927 22:11:59.737207 14043 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43 [sys.catalog]: This master's current role is: LEADER
I20260927 22:11:59.743049 14058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06f3599e94984a84bc1b34dd8154ced0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc33ade617ea4b1cbb78715696ea5f43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f3599e94984a84bc1b34dd8154ced0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } } peers { permanent_uuid: "f4c3c3852bd04b8ca8fed047c48d23f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } } peers { permanent_uuid: "bc33ade617ea4b1cbb78715696ea5f43" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } } }
I20260927 22:11:59.743355 14058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06f3599e94984a84bc1b34dd8154ced0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:11:59.744671 14089 mvcc.cc:205] Tried to move back new op lower bound from 7334081002170261504 to 7334081002014470144. Current Snapshot: MvccSnapshot[applied={T|T < 7334081002170261504}]
I20260927 22:11:59.752301 14087 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:11:59.754469 14087 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:11:59.758139 14072 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c3c3852bd04b8ca8fed047c48d23f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc33ade617ea4b1cbb78715696ea5f43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f3599e94984a84bc1b34dd8154ced0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } } peers { permanent_uuid: "f4c3c3852bd04b8ca8fed047c48d23f9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36111 } } peers { permanent_uuid: "bc33ade617ea4b1cbb78715696ea5f43" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33197 } } }
I20260927 22:11:59.758442 14072 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c3c3852bd04b8ca8fed047c48d23f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:11:59.798753 14087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc33ade617ea4b1cbb78715696ea5f43: Generated new TSK 0
I20260927 22:11:59.800064 14087 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:12:00.474515 14069 catalog_manager.cc:1259] Loaded cluster ID: f2b94c6499d5406982968c7f8a4b3680
I20260927 22:12:00.474687 14069 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 06f3599e94984a84bc1b34dd8154ced0: loading cluster ID for follower catalog manager: success
I20260927 22:12:00.477185 14069 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 06f3599e94984a84bc1b34dd8154ced0: acquiring CA information for follower catalog manager: success
I20260927 22:12:00.479058 14069 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 06f3599e94984a84bc1b34dd8154ced0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:12:00.580413 14083 catalog_manager.cc:1259] Loaded cluster ID: f2b94c6499d5406982968c7f8a4b3680
I20260927 22:12:00.580564 14083 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f4c3c3852bd04b8ca8fed047c48d23f9: loading cluster ID for follower catalog manager: success
I20260927 22:12:00.582943 14083 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f4c3c3852bd04b8ca8fed047c48d23f9: acquiring CA information for follower catalog manager: success
I20260927 22:12:00.584695 14083 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f4c3c3852bd04b8ca8fed047c48d23f9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:12:00.950667 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 13815
I20260927 22:12:00.952996 13877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:12:01.063376 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 13815
I20260927 22:12:01.086282 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 13889
I20260927 22:12:01.087543 13951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:12:01.195564 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 13889
I20260927 22:12:01.216413 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 13963
I20260927 22:12:01.217519 14025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:12:01.327646 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 13963
2026-09-27T22:12:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4866 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:12:01 dist-test-slave-mp03 krb5kdc[14107](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:12:01 dist-test-slave-mp03 krb5kdc[14107](info): set up 2 sockets
Sep 27 22:12:01 dist-test-slave-mp03 krb5kdc[14107](info): commencing operation
krb5kdc: starting...
W20260927 22:12:02.439356 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:12:02 dist-test-slave-mp03 krb5kdc[14107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:12:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:12:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:12:02.683429 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44593
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:12:03.023958 14120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:12:03.024256 14120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:12:03.024425 14120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:12:03.031934 14120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:12:03.032068 14120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:12:03.032177 14120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:12:03.032290 14120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:12:03.032356 14120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:12:03.042924 14120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:12:03.044376 14120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:12:03.045907 14120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:12:03.058421 14125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:03.058368 14128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:03.058449 14126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:12:03.059186 14120 server_base.cc:1034] running on GCE node
I20260927 22:12:03.060058 14120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:12:03.061547 14120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:12:03.062777 14120 hybrid_clock.cc:648] HybridClock initialized: now 1790547123062711 us; error 59 us; skew 500 ppm
I20260927 22:12:03.063158 14120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:12:03 dist-test-slave-mp03 krb5kdc[14107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:12:03.066475 14120 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:12:03.068186 14120 webserver.cc:460] Webserver started at http://127.0.97.62:46861/ using document root <none> and password file <none>
I20260927 22:12:03.068923 14120 fs_manager.cc:359] Metadata directory not provided
I20260927 22:12:03.069077 14120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:12:03.069407 14120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:12:03.072644 14120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "574b49598124474c90693c3340489c14"
format_stamp: "Formatted at 2026-09-27 22:12:03 on dist-test-slave-mp03"
server_key: "c7c8525af8ef9144a5a1b279bae28009"
server_key_iv: "3e874548ea38275d358d95d61aa89cd8"
server_key_version: "encryptionkey@0"
I20260927 22:12:03.073508 14120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "574b49598124474c90693c3340489c14"
format_stamp: "Formatted at 2026-09-27 22:12:03 on dist-test-slave-mp03"
server_key: "c7c8525af8ef9144a5a1b279bae28009"
server_key_iv: "3e874548ea38275d358d95d61aa89cd8"
server_key_version: "encryptionkey@0"
I20260927 22:12:03.082381 14120 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.010s sys 0.000s
I20260927 22:12:03.086357 14135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:03.087920 14120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:12:03.088141 14120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "574b49598124474c90693c3340489c14"
format_stamp: "Formatted at 2026-09-27 22:12:03 on dist-test-slave-mp03"
server_key: "c7c8525af8ef9144a5a1b279bae28009"
server_key_iv: "3e874548ea38275d358d95d61aa89cd8"
server_key_version: "encryptionkey@0"
I20260927 22:12:03.088387 14120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:12:03.107439 14120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:12:03.112128 14120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:12:03.112504 14120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:12:03.133803 14120 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44593
I20260927 22:12:03.133844 14186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44593 every 8 connection(s)
I20260927 22:12:03.135376 14120 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:12:03.137293 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 14120
I20260927 22:12:03.137517 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:12:03.138350 388 external_mini_cluster.cc:1428] Setting key ede27870d2c5bb6e8f8b985390c8aa23
I20260927 22:12:03.140622 14187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:12:03 dist-test-slave-mp03 krb5kdc[14107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:12:03.154955 14187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 574b49598124474c90693c3340489c14: Bootstrap starting.
I20260927 22:12:03.159337 14187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 574b49598124474c90693c3340489c14: Neither blocks nor log segments found. Creating new log.
I20260927 22:12:03.160604 14187 log.cc:824] T 00000000000000000000000000000000 P 574b49598124474c90693c3340489c14: Log is configured to *not* fsync() on all Append() calls
I20260927 22:12:03.163702 14187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 574b49598124474c90693c3340489c14: No bootstrap required, opened a new log
I20260927 22:12:03.164556 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:03.140353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33890 (local address 127.0.97.62:44593)
0927 22:12:03.141204 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:03.141220 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:03.141274 (+ 54us) server_negotiation.cc:407] Connection header received
0927 22:12:03.143610 (+ 2336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:03.143640 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:03.144218 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:03.144674 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:03.146311 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:03.148386 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:03.149940 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:03.151075 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:03.151287 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:03.155912 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:03.155923 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:03.155944 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:03.155998 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:03.160087 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:03.160940 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:03.160947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:03.160954 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:03.161161 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:03.161582 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:03.161587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:03.161591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:03.162116 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:03.162500 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:03.163043 (+ 543us) server_negotiation.cc:299] Negotiation successful
0927 22:12:03.163474 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":373,"threads_started":1}
I20260927 22:12:03.168606 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:12:03.171976 14187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 574b49598124474c90693c3340489c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574b49598124474c90693c3340489c14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44593 } }
I20260927 22:12:03.172374 14187 raft_consensus.cc:374] T 00000000000000000000000000000000 P 574b49598124474c90693c3340489c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:12:03.172484 14187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 574b49598124474c90693c3340489c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 574b49598124474c90693c3340489c14, State: Initialized, Role: FOLLOWER
I20260927 22:12:03.173100 14187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 574b49598124474c90693c3340489c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574b49598124474c90693c3340489c14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44593 } }
I20260927 22:12:03.173337 14187 raft_consensus.cc:388] T 00000000000000000000000000000000 P 574b49598124474c90693c3340489c14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:12:03.173493 14187 raft_consensus.cc:482] T 00000000000000000000000000000000 P 574b49598124474c90693c3340489c14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:12:03.173678 14187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 574b49598124474c90693c3340489c14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:12:03.175928 14187 raft_consensus.cc:504] T 00000000000000000000000000000000 P 574b49598124474c90693c3340489c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574b49598124474c90693c3340489c14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44593 } }
I20260927 22:12:03.176429 14187 leader_election.cc:302] T 00000000000000000000000000000000 P 574b49598124474c90693c3340489c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 574b49598124474c90693c3340489c14; no voters:
I20260927 22:12:03.177033 14187 leader_election.cc:288] T 00000000000000000000000000000000 P 574b49598124474c90693c3340489c14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:12:03.177183 14192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 574b49598124474c90693c3340489c14 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:12:03.177893 14192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 574b49598124474c90693c3340489c14 [term 1 LEADER]: Becoming Leader. State: Replica: 574b49598124474c90693c3340489c14, State: Running, Role: LEADER
I20260927 22:12:03.178442 14192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 574b49598124474c90693c3340489c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574b49598124474c90693c3340489c14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44593 } }
I20260927 22:12:03.179164 14187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 574b49598124474c90693c3340489c14 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:12:03.181313 14193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 574b49598124474c90693c3340489c14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "574b49598124474c90693c3340489c14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574b49598124474c90693c3340489c14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44593 } } }
I20260927 22:12:03.181325 14194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 574b49598124474c90693c3340489c14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 574b49598124474c90693c3340489c14. Latest consensus state: current_term: 1 leader_uuid: "574b49598124474c90693c3340489c14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574b49598124474c90693c3340489c14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44593 } } }
I20260927 22:12:03.181775 14193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 574b49598124474c90693c3340489c14 [sys.catalog]: This master's current role is: LEADER
I20260927 22:12:03.181775 14194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 574b49598124474c90693c3340489c14 [sys.catalog]: This master's current role is: LEADER
W20260927 22:12:03.190448 14208 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 574b49598124474c90693c3340489c14: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:12:03.190585 14208 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:12:03.190807 14199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:12:03.192703 14199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:12:03.201821 14199 catalog_manager.cc:1347] Generated new cluster ID: e2513e5494304cd2bd6d7d3f0a448700
I20260927 22:12:03.201954 14199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:12:03.229200 14199 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:12:03.231336 14199 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:12:03.242627 14199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 574b49598124474c90693c3340489c14: Generated new TSK 0
I20260927 22:12:03.243556 14199 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:12:03.288869 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 14120
I20260927 22:12:03.290151 14182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:12:03.389699 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 14120
2026-09-27T22:12:03Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2058 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:12:03 dist-test-slave-mp03 krb5kdc[14215](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:12:03 dist-test-slave-mp03 krb5kdc[14215](info): set up 2 sockets
Sep 27 22:12:03 dist-test-slave-mp03 krb5kdc[14215](info): commencing operation
krb5kdc: starting...
W20260927 22:12:04.503247 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:12:04 dist-test-slave-mp03 krb5kdc[14215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:12:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:12:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:12:04.747241 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42769
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42769
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:12:05.089125 14228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:12:05.089426 14228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:12:05.089574 14228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:12:05.097117 14228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:12:05.097237 14228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:12:05.097330 14228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:12:05.097431 14228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:12:05.097509 14228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:12:05.107887 14228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:12:05.109423 14228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:12:05.110837 14228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:12:05.123152 14236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:05.123390 14234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:05.123514 14233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:12:05.124919 14228 server_base.cc:1034] running on GCE node
I20260927 22:12:05.125759 14228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:12:05.127425 14228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:12:05.128683 14228 hybrid_clock.cc:648] HybridClock initialized: now 1790547125128628 us; error 37 us; skew 500 ppm
I20260927 22:12:05.129079 14228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:12:05 dist-test-slave-mp03 krb5kdc[14215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:12:05.132368 14228 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:12:05.133996 14228 webserver.cc:460] Webserver started at http://127.0.97.62:40201/ using document root <none> and password file <none>
I20260927 22:12:05.134830 14228 fs_manager.cc:359] Metadata directory not provided
I20260927 22:12:05.134994 14228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:12:05.135327 14228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:12:05.138405 14228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "6bf417a069e240edac650dfc6da196db"
format_stamp: "Formatted at 2026-09-27 22:12:05 on dist-test-slave-mp03"
server_key: "a8a3c928021a68c72936fa0c2a3d947d"
server_key_iv: "a20943fca365a6292a0405f1bb0bbb57"
server_key_version: "encryptionkey@0"
I20260927 22:12:05.139299 14228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "6bf417a069e240edac650dfc6da196db"
format_stamp: "Formatted at 2026-09-27 22:12:05 on dist-test-slave-mp03"
server_key: "a8a3c928021a68c72936fa0c2a3d947d"
server_key_iv: "a20943fca365a6292a0405f1bb0bbb57"
server_key_version: "encryptionkey@0"
I20260927 22:12:05.144891 14228 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:12:05.148510 14243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:05.150105 14228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:12:05.150331 14228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "6bf417a069e240edac650dfc6da196db"
format_stamp: "Formatted at 2026-09-27 22:12:05 on dist-test-slave-mp03"
server_key: "a8a3c928021a68c72936fa0c2a3d947d"
server_key_iv: "a20943fca365a6292a0405f1bb0bbb57"
server_key_version: "encryptionkey@0"
I20260927 22:12:05.150558 14228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:12:05.177585 14228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:12:05.182003 14228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:12:05.182318 14228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:12:05.201941 14228 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42769
I20260927 22:12:05.201970 14294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42769 every 8 connection(s)
I20260927 22:12:05.203395 14228 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:12:05.207248 14295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:12:05.210907 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 14228
I20260927 22:12:05.211143 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:12:05.212029 388 external_mini_cluster.cc:1428] Setting key 8289e302283042ed031cd0260017be57
I20260927 22:12:05.220801 14295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db: Bootstrap starting.
I20260927 22:12:05.224758 14295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db: Neither blocks nor log segments found. Creating new log.
I20260927 22:12:05.226095 14295 log.cc:824] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:12:05 dist-test-slave-mp03 krb5kdc[14215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:12:05.229257 14295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db: No bootstrap required, opened a new log
I20260927 22:12:05.235926 14298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:05.214147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59826 (local address 127.0.97.62:42769)
0927 22:12:05.214929 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:05.214941 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:05.214982 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:12:05.216585 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:05.216608 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:05.217018 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:05.217346 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:05.219134 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:05.220419 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:05.222238 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:05.223036 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:05.223167 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:05.228349 (+ 5182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:05.228357 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:05.228370 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:05.228407 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:05.231693 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:05.232496 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:05.232502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:05.232508 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:05.232676 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:05.233088 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:05.233094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:05.233096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:05.233628 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:05.233872 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:05.234375 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 22:12:05.234859 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":404,"threads_started":1}
I20260927 22:12:05.238557 14295 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf417a069e240edac650dfc6da196db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42769 } }
I20260927 22:12:05.239059 14295 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:12:05.239180 14295 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bf417a069e240edac650dfc6da196db, State: Initialized, Role: FOLLOWER
I20260927 22:12:05.239761 14295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf417a069e240edac650dfc6da196db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42769 } }
I20260927 22:12:05.239957 14295 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:12:05.240134 14295 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:12:05.240315 14295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:12:05.242509 14295 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf417a069e240edac650dfc6da196db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42769 } }
I20260927 22:12:05.243074 14295 leader_election.cc:302] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bf417a069e240edac650dfc6da196db; no voters:
I20260927 22:12:05.243758 14295 leader_election.cc:288] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:12:05.243942 14300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:12:05.244664 14300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [term 1 LEADER]: Becoming Leader. State: Replica: 6bf417a069e240edac650dfc6da196db, State: Running, Role: LEADER
I20260927 22:12:05.245330 14300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf417a069e240edac650dfc6da196db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42769 } }
I20260927 22:12:05.246052 14295 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:12:05.248756 14301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bf417a069e240edac650dfc6da196db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf417a069e240edac650dfc6da196db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42769 } } }
I20260927 22:12:05.249171 14301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [sys.catalog]: This master's current role is: LEADER
I20260927 22:12:05.249338 14302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bf417a069e240edac650dfc6da196db. Latest consensus state: current_term: 1 leader_uuid: "6bf417a069e240edac650dfc6da196db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf417a069e240edac650dfc6da196db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42769 } } }
I20260927 22:12:05.249694 14302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [sys.catalog]: This master's current role is: LEADER
I20260927 22:12:05.251569 14309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:12:05.256421 14309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:12:05.265329 14309 catalog_manager.cc:1347] Generated new cluster ID: 559aee8a582e415f9d4d686f910098f6
I20260927 22:12:05.265456 14309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:12:05.280588 14309 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:12:05.282052 14309 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:12:05.291687 14309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db: Generated new TSK 0
I20260927 22:12:05.292889 14309 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:12:05.359652 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42769
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:12:05.701535 14321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:12:05.701923 14321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:12:05.702023 14321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:12:05.709208 14321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:12:05.709369 14321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:12:05.709489 14321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:12:05.719398 14321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:12:05.720718 14321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:12:05.722096 14321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:12:05.735718 14326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:05.735746 14329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:12:05.736958 14321 server_base.cc:1034] running on GCE node
W20260927 22:12:05.740003 14327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:12:05.740836 14321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:12:05.741803 14321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:12:05.743050 14321 hybrid_clock.cc:648] HybridClock initialized: now 1790547125742972 us; error 79 us; skew 500 ppm
I20260927 22:12:05.743422 14321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:12:05 dist-test-slave-mp03 krb5kdc[14215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:12:05.746986 14321 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:12:05.748823 14321 webserver.cc:460] Webserver started at http://127.0.97.1:45427/ using document root <none> and password file <none>
I20260927 22:12:05.749523 14321 fs_manager.cc:359] Metadata directory not provided
I20260927 22:12:05.749687 14321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:12:05.750072 14321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:12:05.753367 14321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "ab475a17dfe1486b827647383e529d22"
format_stamp: "Formatted at 2026-09-27 22:12:05 on dist-test-slave-mp03"
server_key: "d8ac84e7dbdcb0a2cdfe30346199521c"
server_key_iv: "75cbc51e1d542fa1786ccea638b27137"
server_key_version: "encryptionkey@0"
I20260927 22:12:05.754369 14321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ab475a17dfe1486b827647383e529d22"
format_stamp: "Formatted at 2026-09-27 22:12:05 on dist-test-slave-mp03"
server_key: "d8ac84e7dbdcb0a2cdfe30346199521c"
server_key_iv: "75cbc51e1d542fa1786ccea638b27137"
server_key_version: "encryptionkey@0"
I20260927 22:12:05.760083 14321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:12:05.763808 14336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:05.765267 14321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:12:05.765488 14321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "ab475a17dfe1486b827647383e529d22"
format_stamp: "Formatted at 2026-09-27 22:12:05 on dist-test-slave-mp03"
server_key: "d8ac84e7dbdcb0a2cdfe30346199521c"
server_key_iv: "75cbc51e1d542fa1786ccea638b27137"
server_key_version: "encryptionkey@0"
I20260927 22:12:05.765705 14321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:12:05.779920 14321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:12:05.784606 14321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:12:05.784924 14321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:12:05.785918 14321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:12:05.787318 14321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:12:05.787436 14321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:05.787595 14321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:12:05.787679 14321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:05.822781 14321 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40449
I20260927 22:12:05.822868 14448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40449 every 8 connection(s)
I20260927 22:12:05.824191 14321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:12:05.834416 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 14321
I20260927 22:12:05.834702 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:12:05.835350 388 external_mini_cluster.cc:1428] Setting key f286aecdf1f69a88e7d41a1e4bb37836
Sep 27 22:12:05 dist-test-slave-mp03 krb5kdc[14215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:12:05.846190 14451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:05.827094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37371 (local address 127.0.97.62:42769)
0927 22:12:05.827677 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:05.827683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:05.828514 (+ 831us) server_negotiation.cc:407] Connection header received
0927 22:12:05.830472 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:05.830478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:05.830575 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:05.830774 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:05.832359 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:05.833099 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:05.834918 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:05.835336 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:05.835501 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:05.839745 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:05.839751 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:05.839755 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:05.839800 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:05.842757 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:05.843572 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:05.843578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:05.843582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:05.843723 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:05.844238 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:05.844244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:05.844248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:05.844580 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:05.844757 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:05.845559 (+ 802us) server_negotiation.cc:299] Negotiation successful
0927 22:12:05.845806 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":335,"threads_started":1}
I20260927 22:12:05.847443 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:05.827566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.1:37371)
0927 22:12:05.828334 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:05.828373 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:05.830161 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:05.831030 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:05.831040 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:05.831566 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:12:05.832164 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:05.832181 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:05.833278 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:05.833289 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:05.834749 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:05.834764 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:05.835614 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:05.835621 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:05.835880 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:05.836640 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:12:05.836665 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:12:05.839462 (+ 2797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:12:05.842933 (+ 3471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:05.842942 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:05.842964 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:05.843431 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:05.843851 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:05.843859 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:05.843863 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:05.844093 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:05.844731 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:12:05.844747 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:12:05.845134 (+ 387us) client_negotiation.cc:769] Sending connection context
0927 22:12:05.845449 (+ 315us) client_negotiation.cc:240] Negotiation successful
0927 22:12:05.846052 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":301,"threads_started":1}
I20260927 22:12:05.849478 14449 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42769
I20260927 22:12:05.849838 14449 heartbeater.cc:464] Registering TS with master...
I20260927 22:12:05.850709 14449 heartbeater.cc:511] Master 127.0.97.62:42769 requested a full tablet report, sending...
I20260927 22:12:05.853137 14260 ts_manager.cc:194] Registered new tserver with Master: ab475a17dfe1486b827647383e529d22 (127.0.97.1:40449)
I20260927 22:12:05.855445 14260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37371
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:12:05.891316 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42769
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:12:06.245541 14455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:12:06.245939 14455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:12:06.246227 14455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:12:06.254678 14455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:12:06.254797 14455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:12:06.254935 14455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:12:06.266258 14455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:12:06.267935 14455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:12:06.269472 14455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:12:06.283881 14461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:06.283820 14463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:06.284703 14460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:12:06.285439 14455 server_base.cc:1034] running on GCE node
I20260927 22:12:06.286087 14455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:12:06.287271 14455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:12:06.288522 14455 hybrid_clock.cc:648] HybridClock initialized: now 1790547126288459 us; error 59 us; skew 500 ppm
I20260927 22:12:06.288890 14455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:12:06 dist-test-slave-mp03 krb5kdc[14215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:12:06.292341 14455 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:12:06.293857 14455 webserver.cc:460] Webserver started at http://127.0.97.2:37281/ using document root <none> and password file <none>
I20260927 22:12:06.294641 14455 fs_manager.cc:359] Metadata directory not provided
I20260927 22:12:06.294793 14455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:12:06.295131 14455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:12:06.298125 14455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "ccb5cf86772c48a5a859c0d18cdc200f"
format_stamp: "Formatted at 2026-09-27 22:12:06 on dist-test-slave-mp03"
server_key: "ce362b06f1d75dcc2a4ad2307ab553a8"
server_key_iv: "d57ac84eab760d1d20a5a14d18c7d209"
server_key_version: "encryptionkey@0"
I20260927 22:12:06.298956 14455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ccb5cf86772c48a5a859c0d18cdc200f"
format_stamp: "Formatted at 2026-09-27 22:12:06 on dist-test-slave-mp03"
server_key: "ce362b06f1d75dcc2a4ad2307ab553a8"
server_key_iv: "d57ac84eab760d1d20a5a14d18c7d209"
server_key_version: "encryptionkey@0"
I20260927 22:12:06.305055 14455 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 22:12:06.309224 14470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:06.310866 14455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:12:06.311064 14455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "ccb5cf86772c48a5a859c0d18cdc200f"
format_stamp: "Formatted at 2026-09-27 22:12:06 on dist-test-slave-mp03"
server_key: "ce362b06f1d75dcc2a4ad2307ab553a8"
server_key_iv: "d57ac84eab760d1d20a5a14d18c7d209"
server_key_version: "encryptionkey@0"
I20260927 22:12:06.311352 14455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:12:06.337456 14455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:12:06.341787 14455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:12:06.342094 14455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:12:06.343153 14455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:12:06.344602 14455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:12:06.344674 14455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:06.344789 14455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:12:06.344862 14455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:06.381223 14455 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46467
I20260927 22:12:06.381335 14582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46467 every 8 connection(s)
I20260927 22:12:06.382727 14455 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:12:06.385816 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 14455
I20260927 22:12:06.385993 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:12:06.386442 388 external_mini_cluster.cc:1428] Setting key e41c012cdbfd77e60060f81a509f7982
Sep 27 22:12:06 dist-test-slave-mp03 krb5kdc[14215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:12:06.406668 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:06.385554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37817 (local address 127.0.97.62:42769)
0927 22:12:06.386052 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:06.386058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:06.387158 (+ 1100us) server_negotiation.cc:407] Connection header received
0927 22:12:06.389258 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:06.389267 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:06.389355 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:06.389468 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:06.391102 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:06.392313 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:06.394036 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:06.394857 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:06.395006 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:06.399833 (+ 4827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:06.399839 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:06.399842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:06.399880 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:06.402746 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:06.403587 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:06.403599 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:06.403602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:06.403738 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:06.404220 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:06.404230 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:06.404232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:06.404527 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:06.404690 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:06.405971 (+ 1281us) server_negotiation.cc:299] Negotiation successful
0927 22:12:06.406255 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":245,"threads_started":1}
I20260927 22:12:06.407922 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:06.386091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.2:37817)
0927 22:12:06.386969 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:06.387024 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:06.388974 (+ 1950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:06.389680 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:06.389688 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:06.390265 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:12:06.390921 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:06.390940 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:06.392484 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:06.392488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:06.393835 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:06.393849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:06.395121 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:06.395128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:06.395382 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:06.396216 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:12:06.396327 (+ 111us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:12:06.399541 (+ 3214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:12:06.402929 (+ 3388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:06.402939 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:06.402960 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:06.403437 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:06.403861 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:06.403866 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:06.403870 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:06.404092 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:06.404670 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:12:06.404679 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:12:06.405357 (+ 678us) client_negotiation.cc:769] Sending connection context
0927 22:12:06.405873 (+ 516us) client_negotiation.cc:240] Negotiation successful
0927 22:12:06.406565 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":348,"threads_started":1}
I20260927 22:12:06.409603 14583 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42769
I20260927 22:12:06.409929 14583 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:12:06.410807 14583 heartbeater.cc:511] Master 127.0.97.62:42769 requested a full tablet report, sending...
I20260927 22:12:06.412477 14260 ts_manager.cc:194] Registered new tserver with Master: ccb5cf86772c48a5a859c0d18cdc200f (127.0.97.2:46467)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 22:12:06.413493 14260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37817
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:12:06.441994 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42769
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:12:06.773064 14589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:12:06.773414 14589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:12:06.773582 14589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:12:06.781095 14589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:12:06.781270 14589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:12:06.781462 14589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:12:06.791288 14589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:12:06.792719 14589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:12:06.794122 14589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:12:06.808533 14594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:06.808692 14597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:06.808555 14595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:12:06.809536 14589 server_base.cc:1034] running on GCE node
I20260927 22:12:06.810104 14589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:12:06.811112 14589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:12:06.812352 14589 hybrid_clock.cc:648] HybridClock initialized: now 1790547126812295 us; error 43 us; skew 500 ppm
I20260927 22:12:06.812685 14589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:12:06 dist-test-slave-mp03 krb5kdc[14215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:12:06.819996 14589 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:12:06.821588 14589 webserver.cc:460] Webserver started at http://127.0.97.3:35283/ using document root <none> and password file <none>
I20260927 22:12:06.822391 14589 fs_manager.cc:359] Metadata directory not provided
I20260927 22:12:06.822551 14589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:12:06.822888 14589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:12:06.825860 14589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "730c45f20ad347149922d6f847079c7b"
format_stamp: "Formatted at 2026-09-27 22:12:06 on dist-test-slave-mp03"
server_key: "9296e21010cfbf7e074a3d8fbeb9be8f"
server_key_iv: "de87720459ad3ca945dc44a835fec3dd"
server_key_version: "encryptionkey@0"
I20260927 22:12:06.826812 14589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "730c45f20ad347149922d6f847079c7b"
format_stamp: "Formatted at 2026-09-27 22:12:06 on dist-test-slave-mp03"
server_key: "9296e21010cfbf7e074a3d8fbeb9be8f"
server_key_iv: "de87720459ad3ca945dc44a835fec3dd"
server_key_version: "encryptionkey@0"
I20260927 22:12:06.831957 14589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 22:12:06.835580 14604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:06.837051 14589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:12:06.837248 14589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "730c45f20ad347149922d6f847079c7b"
format_stamp: "Formatted at 2026-09-27 22:12:06 on dist-test-slave-mp03"
server_key: "9296e21010cfbf7e074a3d8fbeb9be8f"
server_key_iv: "de87720459ad3ca945dc44a835fec3dd"
server_key_version: "encryptionkey@0"
I20260927 22:12:06.837481 14589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:12:06.858824 14449 heartbeater.cc:503] Master 127.0.97.62:42769 was elected leader, sending a full tablet report...
I20260927 22:12:06.861435 14589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:12:06.865914 14589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:12:06.866164 14589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:12:06.867177 14589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:12:06.868522 14589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:12:06.868604 14589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:06.868718 14589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:12:06.868826 14589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:06.903358 14589 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36819
I20260927 22:12:06.903455 14716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36819 every 8 connection(s)
I20260927 22:12:06.904785 14589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:12:06.905093 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 14589
I20260927 22:12:06.905267 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:12:06.905974 388 external_mini_cluster.cc:1428] Setting key b8bcc83a3ae595542d6017a5949394a5
Sep 27 22:12:06 dist-test-slave-mp03 krb5kdc[14215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:12:06.927412 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:06.908008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55981 (local address 127.0.97.62:42769)
0927 22:12:06.908607 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:06.908613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:06.909829 (+ 1216us) server_negotiation.cc:407] Connection header received
0927 22:12:06.911955 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:06.911960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:06.912039 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:06.912145 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:06.913806 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:06.914767 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:06.916591 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:06.917044 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:06.917154 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:06.920938 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:06.920966 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:06.920971 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:06.921019 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:06.924070 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:06.924850 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:06.924858 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:06.924862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:06.924995 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:06.925452 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:06.925460 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:06.925464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:06.925759 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:06.925935 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:06.926750 (+ 815us) server_negotiation.cc:299] Negotiation successful
0927 22:12:06.926999 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":326,"threads_started":1}
I20260927 22:12:06.928529 14720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:06.908661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.3:55981)
0927 22:12:06.909648 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:06.909712 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:06.911618 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:06.912368 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:06.912377 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:06.912973 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:12:06.913614 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:06.913639 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:06.914929 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:06.914933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:06.916453 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:06.916466 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:06.917248 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:06.917253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:06.917456 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:06.918219 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:12:06.918243 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:12:06.920696 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:12:06.924234 (+ 3538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:06.924244 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:06.924268 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:06.924720 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:06.925113 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:06.925119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:06.925123 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:06.925331 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:06.925897 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:12:06.925907 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:12:06.926301 (+ 394us) client_negotiation.cc:769] Sending connection context
0927 22:12:06.926635 (+ 334us) client_negotiation.cc:240] Negotiation successful
0927 22:12:06.927254 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":375,"threads_started":1}
I20260927 22:12:06.930047 14717 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42769
I20260927 22:12:06.930437 14717 heartbeater.cc:464] Registering TS with master...
I20260927 22:12:06.931207 14717 heartbeater.cc:511] Master 127.0.97.62:42769 requested a full tablet report, sending...
I20260927 22:12:06.932783 14260 ts_manager.cc:194] Registered new tserver with Master: 730c45f20ad347149922d6f847079c7b (127.0.97.3:36819)
I20260927 22:12:06.933796 14260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55981
I20260927 22:12:06.934731 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:12:06.953310 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:06.940157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59840 (local address 127.0.97.62:42769)
0927 22:12:06.940434 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:06.940440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:06.940858 (+ 418us) server_negotiation.cc:407] Connection header received
0927 22:12:06.941060 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:06.941064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:06.941122 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:06.941238 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:06.942664 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:06.943374 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:06.944996 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:06.945409 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:06.945504 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:06.948053 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:06.948057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:06.948060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:06.948096 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:06.950996 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:06.951646 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:06.951650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:06.951653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:06.951764 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:06.952072 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:06.952075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:06.952077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:06.952399 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:06.952580 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:06.952845 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 22:12:06.953027 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:12:06.957980 14259 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:12:06.965114 14259 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:12:06.989444 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:06.977616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40449 (local address 127.0.0.1:33598)
0927 22:12:06.979379 (+ 1763us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:06.979396 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:06.979595 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:06.980671 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:06.980678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:06.980713 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:12:06.981390 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:06.981405 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:06.983528 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:06.983534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:06.986553 (+ 3019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:06.986567 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:06.988468 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:06.988472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:06.988618 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:06.988654 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:12:06.988793 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 22:12:06.989003 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1647,"mutex_wait_us":246,"thread_start_us":1281,"threads_started":1}
I20260927 22:12:06.990386 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:06.978209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33598 (local address 127.0.97.1:40449)
0927 22:12:06.978982 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:06.978990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:06.979484 (+ 494us) server_negotiation.cc:407] Connection header received
0927 22:12:06.979790 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:06.979802 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:06.980111 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:06.980307 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:12:06.981663 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:06.983371 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:06.986733 (+ 3362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:06.988259 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:06.988393 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:06.988641 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:06.989840 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0927 22:12:06.990020 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":433,"threads_started":1}
I20260927 22:12:06.991468 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:06.977410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46467 (local address 127.0.0.1:43938)
0927 22:12:06.979276 (+ 1866us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:06.979318 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:06.979519 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:06.980628 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:06.980639 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:06.980681 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:12:06.981640 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:06.981649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:06.983528 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:06.983534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:06.986554 (+ 3020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:06.986567 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:06.990537 (+ 3970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:06.990546 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:06.990778 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:06.990818 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 22:12:06.990915 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 22:12:06.991060 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1600,"mutex_wait_us":599,"thread_start_us":2054,"threads_started":1}
I20260927 22:12:06.991513 14731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:06.977560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43938 (local address 127.0.97.2:46467)
0927 22:12:06.978681 (+ 1121us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:06.978694 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:06.979705 (+ 1011us) server_negotiation.cc:407] Connection header received
0927 22:12:06.979815 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:06.979825 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:06.980088 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:06.980285 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:12:06.981824 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:06.983365 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:06.988742 (+ 5377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:06.990026 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:06.990154 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:06.990387 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:06.990974 (+ 587us) server_negotiation.cc:299] Negotiation successful
0927 22:12:06.991158 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":885,"thread_start_us":694,"threads_started":1}
I20260927 22:12:06.991598 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:06.979704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36819 (local address 127.0.0.1:57014)
0927 22:12:06.982834 (+ 3130us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:06.982859 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:06.982991 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:06.984099 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:06.984103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:06.984126 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:12:06.984693 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:06.984702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:06.986545 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:06.986551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:06.988648 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:06.988663 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:06.990768 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:06.990774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:06.990953 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:06.990987 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:12:06.991071 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 22:12:06.991196 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1702,"mutex_wait_us":128,"thread_start_us":1483,"threads_started":1}
I20260927 22:12:06.992211 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:06.980123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57014 (local address 127.0.97.3:36819)
0927 22:12:06.983605 (+ 3482us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:06.983618 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:06.983643 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:12:06.983713 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:06.983718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:06.983897 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:06.984109 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:12:06.984931 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:06.986405 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:06.988814 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:06.990232 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:06.990363 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:06.990621 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:06.991710 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0927 22:12:06.991862 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3337,"thread_start_us":3174,"threads_started":1}
I20260927 22:12:06.995062 14384 tablet_service.cc:1467] Processing CreateTablet for tablet 122900fb557c4c05a1aed57ce62e96ca (DEFAULT_TABLE table=test-table [id=20b93c9a2d794b81aeaae32c2bfe289f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:12:06.995446 14518 tablet_service.cc:1467] Processing CreateTablet for tablet 122900fb557c4c05a1aed57ce62e96ca (DEFAULT_TABLE table=test-table [id=20b93c9a2d794b81aeaae32c2bfe289f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:12:06.996654 14518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122900fb557c4c05a1aed57ce62e96ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:12:06.996686 14384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122900fb557c4c05a1aed57ce62e96ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:12:06.996706 14652 tablet_service.cc:1467] Processing CreateTablet for tablet 122900fb557c4c05a1aed57ce62e96ca (DEFAULT_TABLE table=test-table [id=20b93c9a2d794b81aeaae32c2bfe289f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:12:06.998096 14652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122900fb557c4c05a1aed57ce62e96ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:12:07.009215 14736 tablet_bootstrap.cc:492] T 122900fb557c4c05a1aed57ce62e96ca P ccb5cf86772c48a5a859c0d18cdc200f: Bootstrap starting.
I20260927 22:12:07.011266 14737 tablet_bootstrap.cc:492] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22: Bootstrap starting.
I20260927 22:12:07.012717 14738 tablet_bootstrap.cc:492] T 122900fb557c4c05a1aed57ce62e96ca P 730c45f20ad347149922d6f847079c7b: Bootstrap starting.
I20260927 22:12:07.013146 14736 tablet_bootstrap.cc:654] T 122900fb557c4c05a1aed57ce62e96ca P ccb5cf86772c48a5a859c0d18cdc200f: Neither blocks nor log segments found. Creating new log.
I20260927 22:12:07.014472 14736 log.cc:824] T 122900fb557c4c05a1aed57ce62e96ca P ccb5cf86772c48a5a859c0d18cdc200f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:12:07.014741 14737 tablet_bootstrap.cc:654] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22: Neither blocks nor log segments found. Creating new log.
I20260927 22:12:07.015832 14738 tablet_bootstrap.cc:654] T 122900fb557c4c05a1aed57ce62e96ca P 730c45f20ad347149922d6f847079c7b: Neither blocks nor log segments found. Creating new log.
I20260927 22:12:07.016206 14737 log.cc:824] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22: Log is configured to *not* fsync() on all Append() calls
I20260927 22:12:07.017047 14738 log.cc:824] T 122900fb557c4c05a1aed57ce62e96ca P 730c45f20ad347149922d6f847079c7b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:12:07.017830 14736 tablet_bootstrap.cc:492] T 122900fb557c4c05a1aed57ce62e96ca P ccb5cf86772c48a5a859c0d18cdc200f: No bootstrap required, opened a new log
I20260927 22:12:07.018121 14736 ts_tablet_manager.cc:1397] T 122900fb557c4c05a1aed57ce62e96ca P ccb5cf86772c48a5a859c0d18cdc200f: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260927 22:12:07.019655 14738 tablet_bootstrap.cc:492] T 122900fb557c4c05a1aed57ce62e96ca P 730c45f20ad347149922d6f847079c7b: No bootstrap required, opened a new log
I20260927 22:12:07.019826 14737 tablet_bootstrap.cc:492] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22: No bootstrap required, opened a new log
I20260927 22:12:07.019954 14738 ts_tablet_manager.cc:1397] T 122900fb557c4c05a1aed57ce62e96ca P 730c45f20ad347149922d6f847079c7b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 22:12:07.020073 14737 ts_tablet_manager.cc:1397] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260927 22:12:07.027454 14738 raft_consensus.cc:348] T 122900fb557c4c05a1aed57ce62e96ca P 730c45f20ad347149922d6f847079c7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } } peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } }
I20260927 22:12:07.027827 14738 raft_consensus.cc:374] T 122900fb557c4c05a1aed57ce62e96ca P 730c45f20ad347149922d6f847079c7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:12:07.027967 14738 raft_consensus.cc:729] T 122900fb557c4c05a1aed57ce62e96ca P 730c45f20ad347149922d6f847079c7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730c45f20ad347149922d6f847079c7b, State: Initialized, Role: FOLLOWER
I20260927 22:12:07.028582 14738 consensus_queue.cc:260] T 122900fb557c4c05a1aed57ce62e96ca P 730c45f20ad347149922d6f847079c7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } } peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } }
I20260927 22:12:07.028746 14736 raft_consensus.cc:348] T 122900fb557c4c05a1aed57ce62e96ca P ccb5cf86772c48a5a859c0d18cdc200f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } } peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } }
I20260927 22:12:07.029214 14736 raft_consensus.cc:374] T 122900fb557c4c05a1aed57ce62e96ca P ccb5cf86772c48a5a859c0d18cdc200f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:12:07.029356 14736 raft_consensus.cc:729] T 122900fb557c4c05a1aed57ce62e96ca P ccb5cf86772c48a5a859c0d18cdc200f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb5cf86772c48a5a859c0d18cdc200f, State: Initialized, Role: FOLLOWER
I20260927 22:12:07.029659 14717 heartbeater.cc:503] Master 127.0.97.62:42769 was elected leader, sending a full tablet report...
I20260927 22:12:07.030031 14736 consensus_queue.cc:260] T 122900fb557c4c05a1aed57ce62e96ca P ccb5cf86772c48a5a859c0d18cdc200f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } } peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } }
I20260927 22:12:07.030351 14738 ts_tablet_manager.cc:1428] T 122900fb557c4c05a1aed57ce62e96ca P 730c45f20ad347149922d6f847079c7b: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260927 22:12:07.030392 14737 raft_consensus.cc:348] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } } peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } }
I20260927 22:12:07.030993 14737 raft_consensus.cc:374] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:12:07.031131 14737 raft_consensus.cc:729] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab475a17dfe1486b827647383e529d22, State: Initialized, Role: FOLLOWER
I20260927 22:12:07.031409 14583 heartbeater.cc:503] Master 127.0.97.62:42769 was elected leader, sending a full tablet report...
I20260927 22:12:07.032001 14736 ts_tablet_manager.cc:1428] T 122900fb557c4c05a1aed57ce62e96ca P ccb5cf86772c48a5a859c0d18cdc200f: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260927 22:12:07.031812 14737 consensus_queue.cc:260] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } } peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } }
I20260927 22:12:07.033743 14737 ts_tablet_manager.cc:1428] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
W20260927 22:12:07.076887 14450 tablet.cc:2369] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:12:07.135008 14584 tablet.cc:2369] T 122900fb557c4c05a1aed57ce62e96ca P ccb5cf86772c48a5a859c0d18cdc200f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:12:07.156718 14718 tablet.cc:2369] T 122900fb557c4c05a1aed57ce62e96ca P 730c45f20ad347149922d6f847079c7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:12:07.301903 14744 raft_consensus.cc:482] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:12:07.302211 14744 raft_consensus.cc:504] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } } peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } }
I20260927 22:12:07.303995 14744 leader_election.cc:288] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccb5cf86772c48a5a859c0d18cdc200f (127.0.97.2:46467), 730c45f20ad347149922d6f847079c7b (127.0.97.3:36819)
I20260927 22:12:07.310712 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:07.304451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56749 (local address 127.0.97.3:36819)
0927 22:12:07.304680 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:07.304688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:07.305334 (+ 646us) server_negotiation.cc:407] Connection header received
0927 22:12:07.305550 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:07.305556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:07.305624 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:07.305691 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:12:07.306304 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:07.307084 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:07.308871 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:07.309660 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:07.309753 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:07.309860 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:07.310286 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 22:12:07.310422 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 22:12:07.310719 14731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:07.304390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55645 (local address 127.0.97.2:46467)
0927 22:12:07.304609 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:07.304616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:07.305391 (+ 775us) server_negotiation.cc:407] Connection header received
0927 22:12:07.305585 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:07.305589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:07.305645 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:07.305749 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:12:07.306434 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:07.307198 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:07.308956 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:07.309735 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:07.309874 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:07.309967 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:07.310305 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 22:12:07.310440 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:12:07.311008 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:07.304312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36819 (local address 127.0.97.1:56749)
0927 22:12:07.305205 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:07.305226 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:07.305390 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:07.305736 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:07.305740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:07.305764 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:12:07.306154 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:07.306164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:07.307203 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:07.307208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:07.308715 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:07.308729 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:07.309875 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:07.309881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:07.310064 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:07.310154 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 22:12:07.310227 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 22:12:07.310372 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":55,"thread_start_us":474,"threads_started":1}
I20260927 22:12:07.311087 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:07.304323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46467 (local address 127.0.97.1:55645)
0927 22:12:07.305245 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:07.305263 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:07.305421 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:07.305787 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:07.305791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:07.305819 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:12:07.306264 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:07.306278 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:07.307334 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:07.307337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:07.308761 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:07.308773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:07.309979 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:07.309983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:07.310109 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:07.310163 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 22:12:07.310267 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 22:12:07.310388 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":39,"thread_start_us":641,"threads_started":1}
I20260927 22:12:07.312078 14672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122900fb557c4c05a1aed57ce62e96ca" candidate_uuid: "ab475a17dfe1486b827647383e529d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730c45f20ad347149922d6f847079c7b" is_pre_election: true
I20260927 22:12:07.312633 14672 raft_consensus.cc:2454] T 122900fb557c4c05a1aed57ce62e96ca P 730c45f20ad347149922d6f847079c7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab475a17dfe1486b827647383e529d22 in term 0.
I20260927 22:12:07.312863 14538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122900fb557c4c05a1aed57ce62e96ca" candidate_uuid: "ab475a17dfe1486b827647383e529d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" is_pre_election: true
I20260927 22:12:07.313393 14340 leader_election.cc:302] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730c45f20ad347149922d6f847079c7b, ab475a17dfe1486b827647383e529d22; no voters:
I20260927 22:12:07.313455 14538 raft_consensus.cc:2454] T 122900fb557c4c05a1aed57ce62e96ca P ccb5cf86772c48a5a859c0d18cdc200f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab475a17dfe1486b827647383e529d22 in term 0.
I20260927 22:12:07.313884 14744 raft_consensus.cc:2781] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:12:07.314064 14744 raft_consensus.cc:482] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:12:07.314181 14744 raft_consensus.cc:3037] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:12:07.316761 14744 raft_consensus.cc:504] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } } peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } }
I20260927 22:12:07.317363 14744 leader_election.cc:288] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 1 election: Requested vote from peers ccb5cf86772c48a5a859c0d18cdc200f (127.0.97.2:46467), 730c45f20ad347149922d6f847079c7b (127.0.97.3:36819)
I20260927 22:12:07.317875 14538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122900fb557c4c05a1aed57ce62e96ca" candidate_uuid: "ab475a17dfe1486b827647383e529d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb5cf86772c48a5a859c0d18cdc200f"
I20260927 22:12:07.317898 14672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122900fb557c4c05a1aed57ce62e96ca" candidate_uuid: "ab475a17dfe1486b827647383e529d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730c45f20ad347149922d6f847079c7b"
I20260927 22:12:07.318089 14538 raft_consensus.cc:3037] T 122900fb557c4c05a1aed57ce62e96ca P ccb5cf86772c48a5a859c0d18cdc200f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:12:07.318125 14672 raft_consensus.cc:3037] T 122900fb557c4c05a1aed57ce62e96ca P 730c45f20ad347149922d6f847079c7b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:12:07.320494 14538 raft_consensus.cc:2454] T 122900fb557c4c05a1aed57ce62e96ca P ccb5cf86772c48a5a859c0d18cdc200f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab475a17dfe1486b827647383e529d22 in term 1.
I20260927 22:12:07.320492 14672 raft_consensus.cc:2454] T 122900fb557c4c05a1aed57ce62e96ca P 730c45f20ad347149922d6f847079c7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab475a17dfe1486b827647383e529d22 in term 1.
I20260927 22:12:07.320930 14340 leader_election.cc:302] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730c45f20ad347149922d6f847079c7b, ab475a17dfe1486b827647383e529d22; no voters:
I20260927 22:12:07.321221 14744 raft_consensus.cc:2781] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:12:07.321662 14744 raft_consensus.cc:686] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22 [term 1 LEADER]: Becoming Leader. State: Replica: ab475a17dfe1486b827647383e529d22, State: Running, Role: LEADER
I20260927 22:12:07.322216 14744 consensus_queue.cc:237] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } } peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } }
I20260927 22:12:07.327046 14258 catalog_manager.cc:5495] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab475a17dfe1486b827647383e529d22 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "ab475a17dfe1486b827647383e529d22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } health_report { overall_health: UNKNOWN } } }
I20260927 22:12:07.364194 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:07.358448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33614 (local address 127.0.97.1:40449)
0927 22:12:07.358736 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:07.358742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:07.358763 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:12:07.358991 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:07.358995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:07.359096 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:07.359267 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:12:07.359690 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:07.360467 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:07.361948 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:07.362371 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:07.362507 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:07.363051 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:12:07.363587 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:12:07.363758 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:07.363823 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 22:12:07.363905 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 22:12:07.372519 14672 raft_consensus.cc:1261] T 122900fb557c4c05a1aed57ce62e96ca P 730c45f20ad347149922d6f847079c7b [term 1 FOLLOWER]: Refusing update from remote peer ab475a17dfe1486b827647383e529d22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:12:07.372522 14538 raft_consensus.cc:1261] T 122900fb557c4c05a1aed57ce62e96ca P ccb5cf86772c48a5a859c0d18cdc200f [term 1 FOLLOWER]: Refusing update from remote peer ab475a17dfe1486b827647383e529d22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:12:07.373575 14744 consensus_queue.cc:1035] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:12:07.373879 14747 consensus_queue.cc:1035] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:12:07.387341 14750 mvcc.cc:205] Tried to move back new op lower bound from 7334081033707483136 to 7334081033515757568. Current Snapshot: MvccSnapshot[applied={T|T < 7334081033707483136}]
I20260927 22:12:07.389886 14751 mvcc.cc:205] Tried to move back new op lower bound from 7334081033707483136 to 7334081033515757568. Current Snapshot: MvccSnapshot[applied={T|T < 7334081033707483136}]
I20260927 22:12:07.396402 14752 mvcc.cc:205] Tried to move back new op lower bound from 7334081033707483136 to 7334081033515757568. Current Snapshot: MvccSnapshot[applied={T|T < 7334081033707483136}]
I20260927 22:12:07.408069 14259 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:12:07.408388 14259 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:12:07.412482 14259 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db: Sending DeleteTablet for 3 replicas of tablet 122900fb557c4c05a1aed57ce62e96ca
I20260927 22:12:07.413769 14518 tablet_service.cc:1514] Processing DeleteTablet for tablet 122900fb557c4c05a1aed57ce62e96ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:12:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43938
I20260927 22:12:07.413883 14652 tablet_service.cc:1514] Processing DeleteTablet for tablet 122900fb557c4c05a1aed57ce62e96ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:12:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57014
I20260927 22:12:07.414222 14384 tablet_service.cc:1514] Processing DeleteTablet for tablet 122900fb557c4c05a1aed57ce62e96ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:12:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33598
I20260927 22:12:07.414920 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 14321
I20260927 22:12:07.426138 14762 tablet_replica.cc:330] T 122900fb557c4c05a1aed57ce62e96ca P 730c45f20ad347149922d6f847079c7b: stopping tablet replica
I20260927 22:12:07.426692 14762 raft_consensus.cc:2229] T 122900fb557c4c05a1aed57ce62e96ca P 730c45f20ad347149922d6f847079c7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:12:07.427439 14762 raft_consensus.cc:2258] T 122900fb557c4c05a1aed57ce62e96ca P 730c45f20ad347149922d6f847079c7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:12:07.428618 14763 tablet_replica.cc:330] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22: stopping tablet replica
I20260927 22:12:07.429054 14763 raft_consensus.cc:2229] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:12:07.429769 14763 raft_consensus.cc:2258] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:12:07.430465 14762 ts_tablet_manager.cc:1905] T 122900fb557c4c05a1aed57ce62e96ca P 730c45f20ad347149922d6f847079c7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:12:07.430879 14761 tablet_replica.cc:330] T 122900fb557c4c05a1aed57ce62e96ca P ccb5cf86772c48a5a859c0d18cdc200f: stopping tablet replica
I20260927 22:12:07.431396 14761 raft_consensus.cc:2229] T 122900fb557c4c05a1aed57ce62e96ca P ccb5cf86772c48a5a859c0d18cdc200f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:12:07.432009 14761 raft_consensus.cc:2258] T 122900fb557c4c05a1aed57ce62e96ca P ccb5cf86772c48a5a859c0d18cdc200f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:12:07.432156 14763 ts_tablet_manager.cc:1905] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:12:07.434118 14761 ts_tablet_manager.cc:1905] T 122900fb557c4c05a1aed57ce62e96ca P ccb5cf86772c48a5a859c0d18cdc200f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:12:07.439018 14762 ts_tablet_manager.cc:1918] T 122900fb557c4c05a1aed57ce62e96ca P 730c45f20ad347149922d6f847079c7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:12:07.439224 14762 log.cc:1196] T 122900fb557c4c05a1aed57ce62e96ca P 730c45f20ad347149922d6f847079c7b: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/122900fb557c4c05a1aed57ce62e96ca
I20260927 22:12:07.439774 14762 ts_tablet_manager.cc:1939] T 122900fb557c4c05a1aed57ce62e96ca P 730c45f20ad347149922d6f847079c7b: Deleting consensus metadata
I20260927 22:12:07.440017 14763 ts_tablet_manager.cc:1918] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:12:07.440186 14763 log.cc:1196] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/wal/wals/122900fb557c4c05a1aed57ce62e96ca
I20260927 22:12:07.440737 14763 ts_tablet_manager.cc:1939] T 122900fb557c4c05a1aed57ce62e96ca P ab475a17dfe1486b827647383e529d22: Deleting consensus metadata
I20260927 22:12:07.442071 14247 catalog_manager.cc:4841] TS 730c45f20ad347149922d6f847079c7b (127.0.97.3:36819): tablet 122900fb557c4c05a1aed57ce62e96ca (table test-table [id=20b93c9a2d794b81aeaae32c2bfe289f]) successfully deleted
I20260927 22:12:07.442452 14247 catalog_manager.cc:4841] TS ab475a17dfe1486b827647383e529d22 (127.0.97.1:40449): tablet 122900fb557c4c05a1aed57ce62e96ca (table test-table [id=20b93c9a2d794b81aeaae32c2bfe289f]) successfully deleted
I20260927 22:12:07.442891 14761 ts_tablet_manager.cc:1918] T 122900fb557c4c05a1aed57ce62e96ca P ccb5cf86772c48a5a859c0d18cdc200f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:12:07.443084 14761 log.cc:1196] T 122900fb557c4c05a1aed57ce62e96ca P ccb5cf86772c48a5a859c0d18cdc200f: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/122900fb557c4c05a1aed57ce62e96ca
I20260927 22:12:07.443580 14761 ts_tablet_manager.cc:1939] T 122900fb557c4c05a1aed57ce62e96ca P ccb5cf86772c48a5a859c0d18cdc200f: Deleting consensus metadata
I20260927 22:12:07.445093 14244 catalog_manager.cc:4841] TS ccb5cf86772c48a5a859c0d18cdc200f (127.0.97.2:46467): tablet 122900fb557c4c05a1aed57ce62e96ca (table test-table [id=20b93c9a2d794b81aeaae32c2bfe289f]) successfully deleted
Sep 27 22:12:07 dist-test-slave-mp03 krb5kdc[14215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:12:07.458679 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:07.416384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33616 (local address 127.0.97.1:40449)
0927 22:12:07.416632 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:07.416641 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:07.429924 (+ 13283us) server_negotiation.cc:407] Connection header received
0927 22:12:07.430304 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:07.430309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:07.430400 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:07.430499 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:07.436788 (+ 6289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:07.438168 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:07.440449 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:07.441269 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:07.441374 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:07.451337 (+ 9963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:07.451350 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:07.451367 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:07.451417 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:07.455181 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:07.456100 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:07.456107 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:07.456110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:07.456335 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:07.456798 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:07.456804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:07.456819 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:07.457235 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:07.457560 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:07.457935 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 22:12:07.458207 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:12:07.459425 14444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:12:07.560326 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 14321
I20260927 22:12:07.578392 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 14455
Sep 27 22:12:07 dist-test-slave-mp03 krb5kdc[14215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:12:07.593981 14731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:07.579242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.0.97.2:46467)
0927 22:12:07.579451 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:07.579456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:07.579475 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:12:07.579624 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:07.579628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:07.579691 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:07.579782 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:07.581140 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:07.581885 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:07.583325 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:07.583743 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:07.583857 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:07.588212 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:07.588235 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:07.588254 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:07.588294 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:07.591279 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:07.592002 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:07.592011 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:07.592013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:07.592151 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:07.592511 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:07.592514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:07.592516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:07.592902 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:07.593129 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:07.593420 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 22:12:07.593595 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:12:07.594741 14578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:12:07.686152 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 14455
I20260927 22:12:07.702934 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 14589
Sep 27 22:12:07 dist-test-slave-mp03 krb5kdc[14215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:12:07.719161 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:07.703734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57020 (local address 127.0.97.3:36819)
0927 22:12:07.703917 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:07.703924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:07.704005 (+ 81us) server_negotiation.cc:407] Connection header received
0927 22:12:07.704224 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:07.704228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:07.704371 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:07.704483 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:07.705801 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:07.706515 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:07.707903 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:07.708328 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:07.708467 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:07.712872 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:07.712888 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:07.712906 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:07.712946 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:07.716018 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:07.716821 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:07.716829 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:07.716831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:07.716969 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:07.717367 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:07.717372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:07.717375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:07.717764 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:07.718073 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:07.718402 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 22:12:07.718838 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:12:07.720005 14712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:12:07.809538 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 14589
I20260927 22:12:07.826223 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 14228
I20260927 22:12:07.827533 14290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:12:07.939535 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 14228
I20260927 22:12:07.961495 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42769
--webserver_interface=127.0.97.62
--webserver_port=40201
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:12:08.291003 14769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:12:08.291308 14769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:12:08.291457 14769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:12:08.298731 14769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:12:08.298840 14769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:12:08.298933 14769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:12:08.299028 14769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:12:08.299110 14769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:12:08.309094 14769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=40201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:12:08.310479 14769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:12:08.311856 14769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:12:08.335570 14775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:08.335587 14778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:08.335619 14776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:12:08.337028 14769 server_base.cc:1034] running on GCE node
I20260927 22:12:08.338009 14769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:12:08.339736 14769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:12:08.340974 14769 hybrid_clock.cc:648] HybridClock initialized: now 1790547128340910 us; error 52 us; skew 500 ppm
I20260927 22:12:08.341382 14769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:12:08 dist-test-slave-mp03 krb5kdc[14215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:12:08.345082 14769 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:12:08.346866 14769 webserver.cc:460] Webserver started at http://127.0.97.62:40201/ using document root <none> and password file <none>
I20260927 22:12:08.347671 14769 fs_manager.cc:359] Metadata directory not provided
I20260927 22:12:08.347836 14769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:12:08.354020 14769 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 22:12:08.357393 14785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:08.359086 14769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 22:12:08.359303 14769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "6bf417a069e240edac650dfc6da196db"
format_stamp: "Formatted at 2026-09-27 22:12:05 on dist-test-slave-mp03"
server_key: "a8a3c928021a68c72936fa0c2a3d947d"
server_key_iv: "a20943fca365a6292a0405f1bb0bbb57"
server_key_version: "encryptionkey@0"
I20260927 22:12:08.360467 14769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:12:08.373302 14769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:12:08.377841 14769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:12:08.378135 14769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:12:08.398123 14769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42769
I20260927 22:12:08.398129 14836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42769 every 8 connection(s)
I20260927 22:12:08.399780 14769 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:12:08.404570 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 14769
I20260927 22:12:08.405460 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40449
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42769
--webserver_port=45427
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:12:08.407400 14837 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:12:08.411094 14837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db: Bootstrap starting.
I20260927 22:12:08.430825 14837 log.cc:824] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db: Log is configured to *not* fsync() on all Append() calls
I20260927 22:12:08.443590 14837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:12:08.444175 14837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db: Bootstrap complete.
I20260927 22:12:08.453527 14837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf417a069e240edac650dfc6da196db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42769 } }
I20260927 22:12:08.454331 14837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bf417a069e240edac650dfc6da196db, State: Initialized, Role: FOLLOWER
I20260927 22:12:08.455025 14837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf417a069e240edac650dfc6da196db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42769 } }
I20260927 22:12:08.455273 14837 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:12:08.455392 14837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:12:08.455580 14837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:12:08.459533 14837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf417a069e240edac650dfc6da196db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42769 } }
I20260927 22:12:08.460036 14837 leader_election.cc:302] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bf417a069e240edac650dfc6da196db; no voters:
I20260927 22:12:08.460657 14837 leader_election.cc:288] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:12:08.460800 14841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:12:08.463017 14841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [term 2 LEADER]: Becoming Leader. State: Replica: 6bf417a069e240edac650dfc6da196db, State: Running, Role: LEADER
I20260927 22:12:08.463227 14837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:12:08.463608 14841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf417a069e240edac650dfc6da196db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42769 } }
I20260927 22:12:08.471038 14842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6bf417a069e240edac650dfc6da196db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf417a069e240edac650dfc6da196db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42769 } } }
I20260927 22:12:08.471549 14842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [sys.catalog]: This master's current role is: LEADER
I20260927 22:12:08.472167 14843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bf417a069e240edac650dfc6da196db. Latest consensus state: current_term: 2 leader_uuid: "6bf417a069e240edac650dfc6da196db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf417a069e240edac650dfc6da196db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42769 } } }
I20260927 22:12:08.472486 14843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [sys.catalog]: This master's current role is: LEADER
I20260927 22:12:08.476653 14850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:12:08.484894 14850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:12:08.486325 14850 catalog_manager.cc:1259] Loaded cluster ID: 559aee8a582e415f9d4d686f910098f6
I20260927 22:12:08.486481 14850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:12:08.489542 14850 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 22:12:08.507351 14850 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 22:12:08.516685 14850 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:12:08.518712 14850 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db: Loaded TSK: 0
I20260927 22:12:08.520313 14850 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:12:08.760967 14839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:12:08.761318 14839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:12:08.761478 14839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:12:08.769362 14839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:12:08.769529 14839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:12:08.769704 14839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:12:08.780140 14839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45427
--tserver_master_addrs=127.0.97.62:42769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:12:08.781512 14839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:12:08.783041 14839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:12:08.797432 14866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:08.797436 14865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:08.798183 14868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:12:08.798470 14839 server_base.cc:1034] running on GCE node
I20260927 22:12:08.799192 14839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:12:08.800264 14839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:12:08.801527 14839 hybrid_clock.cc:648] HybridClock initialized: now 1790547128801456 us; error 61 us; skew 500 ppm
I20260927 22:12:08.801891 14839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:12:08 dist-test-slave-mp03 krb5kdc[14215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:12:08.805351 14839 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:12:08.807116 14839 webserver.cc:460] Webserver started at http://127.0.97.1:45427/ using document root <none> and password file <none>
I20260927 22:12:08.808019 14839 fs_manager.cc:359] Metadata directory not provided
I20260927 22:12:08.808190 14839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:12:08.813994 14839 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 22:12:08.816998 14875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:08.818512 14839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:12:08.818710 14839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "ab475a17dfe1486b827647383e529d22"
format_stamp: "Formatted at 2026-09-27 22:12:05 on dist-test-slave-mp03"
server_key: "d8ac84e7dbdcb0a2cdfe30346199521c"
server_key_iv: "75cbc51e1d542fa1786ccea638b27137"
server_key_version: "encryptionkey@0"
I20260927 22:12:08.819705 14839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:12:08.836089 14839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:12:08.841145 14839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:12:08.841454 14839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:12:08.842651 14839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:12:08.844190 14839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:12:08.844286 14839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:08.844444 14839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:12:08.844543 14839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:08.880374 14839 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40449
I20260927 22:12:08.880512 14987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40449 every 8 connection(s)
I20260927 22:12:08.881945 14839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:12:08.889182 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 14839
I20260927 22:12:08.889983 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:46467
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42769
--webserver_port=37281
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:12:08 dist-test-slave-mp03 krb5kdc[14215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:12:08.908617 14991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:08.885318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.1:49457)
0927 22:12:08.886164 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:08.886211 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:08.887972 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:08.889708 (+ 1736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:08.889716 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:08.890378 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:12:08.891012 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:08.891031 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:08.893516 (+ 2485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:08.893520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:08.894884 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:08.894897 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:08.896456 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:08.896462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:08.896648 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:08.897444 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:12:08.897468 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:12:08.900069 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:12:08.903882 (+ 3813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:08.903900 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:08.903925 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:08.904389 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:08.904860 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:08.904866 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:08.904869 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:08.905083 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:08.905870 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:12:08.905879 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:12:08.906262 (+ 383us) client_negotiation.cc:769] Sending connection context
0927 22:12:08.906544 (+ 282us) client_negotiation.cc:240] Negotiation successful
0927 22:12:08.907197 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":359,"threads_started":1}
I20260927 22:12:08.908617 14990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:08.885030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49457 (local address 127.0.97.62:42769)
0927 22:12:08.885593 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:08.885604 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:08.886354 (+ 750us) server_negotiation.cc:407] Connection header received
0927 22:12:08.888592 (+ 2238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:08.888613 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:08.889147 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:08.889570 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:08.891250 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:08.893324 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:08.895028 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:08.896199 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:08.896416 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:08.900441 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:08.900450 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:08.900469 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:08.900517 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:08.903696 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:08.904559 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:08.904566 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:08.904573 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:08.904737 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:08.905214 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:08.905219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:08.905221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:08.905723 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:08.905937 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:08.906809 (+ 872us) server_negotiation.cc:299] Negotiation successful
0927 22:12:08.907243 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":231,"threads_started":1}
I20260927 22:12:08.911895 14988 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42769
I20260927 22:12:08.912288 14988 heartbeater.cc:464] Registering TS with master...
I20260927 22:12:08.913321 14988 heartbeater.cc:511] Master 127.0.97.62:42769 requested a full tablet report, sending...
I20260927 22:12:08.916038 14802 ts_manager.cc:194] Registered new tserver with Master: ab475a17dfe1486b827647383e529d22 (127.0.97.1:40449)
I20260927 22:12:08.917658 14802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49457
W20260927 22:12:09.231238 14992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:12:09.231523 14992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:12:09.231704 14992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:12:09.239285 14992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:12:09.239411 14992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:12:09.239610 14992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:12:09.249562 14992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:46467
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=37281
--tserver_master_addrs=127.0.97.62:42769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:12:09.250913 14992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:12:09.252274 14992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:12:09.265834 14997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:09.265857 15000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:09.265789 14998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:12:09.266793 14992 server_base.cc:1034] running on GCE node
I20260927 22:12:09.267459 14992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:12:09.268378 14992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:12:09.269590 14992 hybrid_clock.cc:648] HybridClock initialized: now 1790547129269535 us; error 43 us; skew 500 ppm
I20260927 22:12:09.269927 14992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:12:09 dist-test-slave-mp03 krb5kdc[14215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:12:09.273069 14992 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:12:09.274515 14992 webserver.cc:460] Webserver started at http://127.0.97.2:37281/ using document root <none> and password file <none>
I20260927 22:12:09.275279 14992 fs_manager.cc:359] Metadata directory not provided
I20260927 22:12:09.275444 14992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:12:09.281073 14992 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 22:12:09.283933 15007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:09.285315 14992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:12:09.285503 14992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "ccb5cf86772c48a5a859c0d18cdc200f"
format_stamp: "Formatted at 2026-09-27 22:12:06 on dist-test-slave-mp03"
server_key: "ce362b06f1d75dcc2a4ad2307ab553a8"
server_key_iv: "d57ac84eab760d1d20a5a14d18c7d209"
server_key_version: "encryptionkey@0"
I20260927 22:12:09.286579 14992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:12:09.312865 14992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:12:09.317629 14992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:12:09.317873 14992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:12:09.318972 14992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:12:09.320420 14992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:12:09.320518 14992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:09.320601 14992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:12:09.320655 14992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:09.355140 14992 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46467
I20260927 22:12:09.355175 15119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46467 every 8 connection(s)
I20260927 22:12:09.357069 14992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:12:09.365276 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 14992
I20260927 22:12:09.365924 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36819
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42769
--webserver_port=35283
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:12:09 dist-test-slave-mp03 krb5kdc[14215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:12:09.380657 14990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:09.359842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33467 (local address 127.0.97.62:42769)
0927 22:12:09.360023 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:09.360028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:09.361314 (+ 1286us) server_negotiation.cc:407] Connection header received
0927 22:12:09.363246 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:09.363251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:09.363347 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:09.363460 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:09.365502 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.366286 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.369110 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.369560 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.369664 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:09.374265 (+ 4601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:09.374271 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:09.374274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:09.374312 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:09.377259 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:09.378211 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:09.378215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:09.378218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:09.378348 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:09.378837 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:09.378841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:09.378843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:09.379124 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:09.379242 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:09.380175 (+ 933us) server_negotiation.cc:299] Negotiation successful
0927 22:12:09.380384 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:12:09.382159 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:09.360278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.2:33467)
0927 22:12:09.361145 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:09.361184 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:09.362959 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:09.363717 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:09.363725 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:09.364660 (+ 935us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:12:09.365270 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:09.365296 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.366447 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.366456 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:09.368936 (+ 2480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:09.368955 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.369815 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.369827 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:09.370194 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:09.371127 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:12:09.371170 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:12:09.374017 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:12:09.377418 (+ 3401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:09.377430 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:09.377452 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:09.378084 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:09.378462 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:09.378470 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:09.378476 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:09.378733 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:09.379252 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:12:09.379263 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:12:09.379740 (+ 477us) client_negotiation.cc:769] Sending connection context
0927 22:12:09.380049 (+ 309us) client_negotiation.cc:240] Negotiation successful
0927 22:12:09.380650 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":439,"threads_started":1}
I20260927 22:12:09.383759 15120 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42769
I20260927 22:12:09.384090 15120 heartbeater.cc:464] Registering TS with master...
I20260927 22:12:09.384824 15120 heartbeater.cc:511] Master 127.0.97.62:42769 requested a full tablet report, sending...
I20260927 22:12:09.386345 14802 ts_manager.cc:194] Registered new tserver with Master: ccb5cf86772c48a5a859c0d18cdc200f (127.0.97.2:46467)
I20260927 22:12:09.387238 14802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33467
W20260927 22:12:09.697594 15123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:12:09.697995 15123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:12:09.698177 15123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:12:09.705828 15123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:12:09.705976 15123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:12:09.706172 15123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:12:09.716468 15123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35283
--tserver_master_addrs=127.0.97.62:42769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:12:09.717984 15123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:12:09.719503 15123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:12:09.733841 15129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:09.733893 15128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:09.733901 15131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:12:09.734798 15123 server_base.cc:1034] running on GCE node
I20260927 22:12:09.735409 15123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:12:09.736390 15123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:12:09.737624 15123 hybrid_clock.cc:648] HybridClock initialized: now 1790547129737579 us; error 68 us; skew 500 ppm
I20260927 22:12:09.737946 15123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:12:09 dist-test-slave-mp03 krb5kdc[14215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:12:09.741120 15123 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:12:09.742688 15123 webserver.cc:460] Webserver started at http://127.0.97.3:35283/ using document root <none> and password file <none>
I20260927 22:12:09.743407 15123 fs_manager.cc:359] Metadata directory not provided
I20260927 22:12:09.743566 15123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:12:09.748937 15123 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 22:12:09.751847 15138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:09.753465 15123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:12:09.753672 15123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "730c45f20ad347149922d6f847079c7b"
format_stamp: "Formatted at 2026-09-27 22:12:06 on dist-test-slave-mp03"
server_key: "9296e21010cfbf7e074a3d8fbeb9be8f"
server_key_iv: "de87720459ad3ca945dc44a835fec3dd"
server_key_version: "encryptionkey@0"
I20260927 22:12:09.754727 15123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:12:09.772858 15123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:12:09.777179 15123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:12:09.777462 15123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:12:09.778427 15123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:12:09.779757 15123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:12:09.779887 15123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:09.780018 15123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:12:09.780112 15123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:09.811877 15123 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36819
I20260927 22:12:09.811957 15250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36819 every 8 connection(s)
I20260927 22:12:09.813446 15123 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:12:09.823402 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 15123
Sep 27 22:12:09 dist-test-slave-mp03 krb5kdc[14215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:12:09.835807 14990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:09.816088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51247 (local address 127.0.97.62:42769)
0927 22:12:09.816285 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:09.816289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:09.817462 (+ 1173us) server_negotiation.cc:407] Connection header received
0927 22:12:09.819389 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:09.819394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:09.819490 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:09.819600 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:09.821079 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.821805 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.823866 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.824400 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.824529 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:09.828745 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:09.828750 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:09.828753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:09.828791 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:09.832035 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:09.832860 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:09.832872 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:09.832874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:09.833009 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:09.833530 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:09.833534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:09.833536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:09.833894 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:09.834075 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:09.835271 (+ 1196us) server_negotiation.cc:299] Negotiation successful
0927 22:12:09.835540 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:12:09.837378 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:09.816560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.3:51247)
0927 22:12:09.817254 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:09.817331 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:09.819107 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:09.819846 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:09.819854 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:09.820357 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:12:09.820905 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:09.820922 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.821977 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.821982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:09.823717 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:09.823732 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.824666 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.824677 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:09.825178 (+ 501us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:09.825902 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:12:09.825927 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:12:09.828512 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:12:09.832197 (+ 3685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:09.832208 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:09.832229 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:09.832714 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:09.833136 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:09.833141 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:09.833145 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:09.833372 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:09.834077 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:12:09.834087 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:12:09.834661 (+ 574us) client_negotiation.cc:769] Sending connection context
0927 22:12:09.835053 (+ 392us) client_negotiation.cc:240] Negotiation successful
0927 22:12:09.835682 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":287,"threads_started":1}
I20260927 22:12:09.839267 15251 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42769
I20260927 22:12:09.839695 15251 heartbeater.cc:464] Registering TS with master...
I20260927 22:12:09.840353 15254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:09.824491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42210 (local address 127.0.97.62:42769)
0927 22:12:09.825053 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:09.825058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:09.825338 (+ 280us) server_negotiation.cc:407] Connection header received
0927 22:12:09.825547 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:09.825550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:09.825619 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:09.825738 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:09.827360 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.828419 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.830041 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.830513 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.830687 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:09.833295 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:09.833300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:09.833303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:09.833337 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:09.836926 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:09.837878 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:09.837881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:09.837884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:09.838006 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:09.838757 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:09.838762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:09.838764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:09.839057 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:09.839294 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:09.839703 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 22:12:09.839957 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":287,"threads_started":1}
I20260927 22:12:09.840612 15251 heartbeater.cc:511] Master 127.0.97.62:42769 requested a full tablet report, sending...
I20260927 22:12:09.842312 14801 ts_manager.cc:194] Registered new tserver with Master: 730c45f20ad347149922d6f847079c7b (127.0.97.3:36819)
I20260927 22:12:09.843340 14801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51247
I20260927 22:12:09.855350 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:12:09.862748 15254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:09.856359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42226 (local address 127.0.97.62:42769)
0927 22:12:09.856542 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:09.856547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:09.857015 (+ 468us) server_negotiation.cc:407] Connection header received
0927 22:12:09.857215 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:09.857219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:09.857355 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:09.857512 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:12:09.857934 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.858763 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.860474 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.860957 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.861052 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:09.861550 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:12:09.862202 (+ 652us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:12:09.862319 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:09.862426 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 22:12:09.862491 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:12:09.863758 14801 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:42226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:12:09.871275 14801 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:12:09.901612 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:09.885105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59530 (local address 127.0.97.2:46467)
0927 22:12:09.887556 (+ 2451us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:09.887564 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:09.888093 (+ 529us) server_negotiation.cc:407] Connection header received
0927 22:12:09.888442 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:09.888450 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:09.888726 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:09.888943 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:12:09.890156 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.892152 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.895701 (+ 3549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.897734 (+ 2033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.897872 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:09.898149 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:09.901047 (+ 2898us) server_negotiation.cc:299] Negotiation successful
0927 22:12:09.901251 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2285,"thread_start_us":2191,"threads_started":1}
I20260927 22:12:09.901705 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:09.884902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46467 (local address 127.0.0.1:59530)
0927 22:12:09.887986 (+ 3084us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:09.888005 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:09.888197 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:09.889302 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:09.889314 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:09.889356 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:12:09.889899 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:09.889913 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.892392 (+ 2479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.892400 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:09.895486 (+ 3086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:09.895501 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.898397 (+ 2896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.898406 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:09.898695 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:09.900801 (+ 2106us) client_negotiation.cc:769] Sending connection context
0927 22:12:09.900972 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 22:12:09.901208 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2966,"mutex_wait_us":3,"thread_start_us":1366,"threads_started":1}
I20260927 22:12:09.905575 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:09.884256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40449 (local address 127.0.0.1:38232)
0927 22:12:09.886791 (+ 2535us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:09.886837 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:09.887011 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:09.898625 (+ 11614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:09.898634 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:09.898677 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:12:09.899125 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:09.899136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.900771 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.900778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:09.902752 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:09.902765 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.904687 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.904694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:09.904889 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:09.904926 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:12:09.904990 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 22:12:09.905142 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2206,"mutex_wait_us":19,"thread_start_us":3480,"threads_started":1}
I20260927 22:12:09.905985 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:09.885425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38232 (local address 127.0.97.1:40449)
0927 22:12:09.897456 (+ 12031us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:09.897470 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:09.897512 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:12:09.897644 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:09.897656 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:09.898051 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:09.898793 (+ 742us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:12:09.899335 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.900619 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.902911 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.904440 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.905132 (+ 692us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:09.905395 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:09.905486 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 22:12:09.905653 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11776,"thread_start_us":12824,"threads_started":1}
I20260927 22:12:09.906678 15055 tablet_service.cc:1467] Processing CreateTablet for tablet 08291b00b0874cbfb55389b50d37ab24 (DEFAULT_TABLE table=test-table [id=8c053ea753434fc2875bfd82e6a1e11c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:12:09.907915 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:09.888453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36819 (local address 127.0.0.1:56328)
0927 22:12:09.899581 (+ 11128us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:09.899605 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:09.899724 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:09.900444 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:09.900449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:09.900473 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:12:09.900948 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:09.900967 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.903194 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.903199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:09.905202 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:09.905214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.907105 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.907111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:09.907309 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:09.907342 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:12:09.907443 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 22:12:09.907580 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10939,"thread_start_us":10794,"threads_started":1}
I20260927 22:12:09.908241 15055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08291b00b0874cbfb55389b50d37ab24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:12:09.908700 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:09.888524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56328 (local address 127.0.97.3:36819)
0927 22:12:09.899935 (+ 11411us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:09.899942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:09.899968 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:12:09.900045 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:09.900052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:09.900245 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:09.900460 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:12:09.901196 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.903047 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.905391 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.906878 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.907030 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:09.907260 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:09.908145 (+ 885us) server_negotiation.cc:299] Negotiation successful
0927 22:12:09.908359 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11222,"thread_start_us":11336,"threads_started":1}
I20260927 22:12:09.909615 14923 tablet_service.cc:1467] Processing CreateTablet for tablet 08291b00b0874cbfb55389b50d37ab24 (DEFAULT_TABLE table=test-table [id=8c053ea753434fc2875bfd82e6a1e11c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:12:09.911036 14923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08291b00b0874cbfb55389b50d37ab24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:12:09.913921 15186 tablet_service.cc:1467] Processing CreateTablet for tablet 08291b00b0874cbfb55389b50d37ab24 (DEFAULT_TABLE table=test-table [id=8c053ea753434fc2875bfd82e6a1e11c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:12:09.915426 15186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08291b00b0874cbfb55389b50d37ab24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:12:09.921069 15267 tablet_bootstrap.cc:492] T 08291b00b0874cbfb55389b50d37ab24 P ccb5cf86772c48a5a859c0d18cdc200f: Bootstrap starting.
I20260927 22:12:09.921559 14988 heartbeater.cc:503] Master 127.0.97.62:42769 was elected leader, sending a full tablet report...
I20260927 22:12:09.925330 15267 tablet_bootstrap.cc:654] T 08291b00b0874cbfb55389b50d37ab24 P ccb5cf86772c48a5a859c0d18cdc200f: Neither blocks nor log segments found. Creating new log.
I20260927 22:12:09.927240 15268 tablet_bootstrap.cc:492] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22: Bootstrap starting.
I20260927 22:12:09.927577 15267 log.cc:824] T 08291b00b0874cbfb55389b50d37ab24 P ccb5cf86772c48a5a859c0d18cdc200f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:12:09.930917 15267 tablet_bootstrap.cc:492] T 08291b00b0874cbfb55389b50d37ab24 P ccb5cf86772c48a5a859c0d18cdc200f: No bootstrap required, opened a new log
I20260927 22:12:09.931214 15267 ts_tablet_manager.cc:1397] T 08291b00b0874cbfb55389b50d37ab24 P ccb5cf86772c48a5a859c0d18cdc200f: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 22:12:09.931324 15268 tablet_bootstrap.cc:654] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22: Neither blocks nor log segments found. Creating new log.
I20260927 22:12:09.932190 15270 tablet_bootstrap.cc:492] T 08291b00b0874cbfb55389b50d37ab24 P 730c45f20ad347149922d6f847079c7b: Bootstrap starting.
I20260927 22:12:09.932621 15268 log.cc:824] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22: Log is configured to *not* fsync() on all Append() calls
I20260927 22:12:09.935470 15270 tablet_bootstrap.cc:654] T 08291b00b0874cbfb55389b50d37ab24 P 730c45f20ad347149922d6f847079c7b: Neither blocks nor log segments found. Creating new log.
I20260927 22:12:09.935877 15268 tablet_bootstrap.cc:492] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22: No bootstrap required, opened a new log
I20260927 22:12:09.936156 15268 ts_tablet_manager.cc:1397] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260927 22:12:09.936553 15270 log.cc:824] T 08291b00b0874cbfb55389b50d37ab24 P 730c45f20ad347149922d6f847079c7b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:12:09.939894 15270 tablet_bootstrap.cc:492] T 08291b00b0874cbfb55389b50d37ab24 P 730c45f20ad347149922d6f847079c7b: No bootstrap required, opened a new log
I20260927 22:12:09.940197 15270 ts_tablet_manager.cc:1397] T 08291b00b0874cbfb55389b50d37ab24 P 730c45f20ad347149922d6f847079c7b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 22:12:09.942405 15267 raft_consensus.cc:348] T 08291b00b0874cbfb55389b50d37ab24 P ccb5cf86772c48a5a859c0d18cdc200f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } } peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } }
I20260927 22:12:09.942914 15267 raft_consensus.cc:374] T 08291b00b0874cbfb55389b50d37ab24 P ccb5cf86772c48a5a859c0d18cdc200f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:12:09.943014 15267 raft_consensus.cc:729] T 08291b00b0874cbfb55389b50d37ab24 P ccb5cf86772c48a5a859c0d18cdc200f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb5cf86772c48a5a859c0d18cdc200f, State: Initialized, Role: FOLLOWER
I20260927 22:12:09.943665 15267 consensus_queue.cc:260] T 08291b00b0874cbfb55389b50d37ab24 P ccb5cf86772c48a5a859c0d18cdc200f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } } peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } }
I20260927 22:12:09.944857 15120 heartbeater.cc:503] Master 127.0.97.62:42769 was elected leader, sending a full tablet report...
I20260927 22:12:09.947047 15267 ts_tablet_manager.cc:1428] T 08291b00b0874cbfb55389b50d37ab24 P ccb5cf86772c48a5a859c0d18cdc200f: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260927 22:12:09.948141 15268 raft_consensus.cc:348] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } } peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } }
I20260927 22:12:09.948896 15268 raft_consensus.cc:374] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:12:09.948676 15270 raft_consensus.cc:348] T 08291b00b0874cbfb55389b50d37ab24 P 730c45f20ad347149922d6f847079c7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } } peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } }
I20260927 22:12:09.949059 15268 raft_consensus.cc:729] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab475a17dfe1486b827647383e529d22, State: Initialized, Role: FOLLOWER
I20260927 22:12:09.949167 15270 raft_consensus.cc:374] T 08291b00b0874cbfb55389b50d37ab24 P 730c45f20ad347149922d6f847079c7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:12:09.949265 15270 raft_consensus.cc:729] T 08291b00b0874cbfb55389b50d37ab24 P 730c45f20ad347149922d6f847079c7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730c45f20ad347149922d6f847079c7b, State: Initialized, Role: FOLLOWER
I20260927 22:12:09.949864 15270 consensus_queue.cc:260] T 08291b00b0874cbfb55389b50d37ab24 P 730c45f20ad347149922d6f847079c7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } } peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } }
I20260927 22:12:09.949916 15268 consensus_queue.cc:260] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } } peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } }
I20260927 22:12:09.952252 15268 ts_tablet_manager.cc:1428] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260927 22:12:09.954253 15251 heartbeater.cc:503] Master 127.0.97.62:42769 was elected leader, sending a full tablet report...
I20260927 22:12:09.954972 15270 ts_tablet_manager.cc:1428] T 08291b00b0874cbfb55389b50d37ab24 P 730c45f20ad347149922d6f847079c7b: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260927 22:12:09.981936 15274 raft_consensus.cc:482] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:12:09.982280 15274 raft_consensus.cc:504] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } } peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } }
I20260927 22:12:09.984223 15274 leader_election.cc:288] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730c45f20ad347149922d6f847079c7b (127.0.97.3:36819), ccb5cf86772c48a5a859c0d18cdc200f (127.0.97.2:46467)
I20260927 22:12:09.991448 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:09.984762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51425 (local address 127.0.97.3:36819)
0927 22:12:09.985023 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:09.985032 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:09.985762 (+ 730us) server_negotiation.cc:407] Connection header received
0927 22:12:09.985975 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:09.985983 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:09.986089 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:09.986208 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:12:09.986939 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.987857 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.989731 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.990504 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.990675 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:09.990781 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:09.991051 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 22:12:09.991175 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 22:12:09.991889 15276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:09.984437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36819 (local address 127.0.97.1:51425)
0927 22:12:09.985635 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:09.985650 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:09.985782 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:09.986233 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:09.986238 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:09.986265 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:12:09.986752 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:09.986765 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.987988 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.987996 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:09.989580 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:09.989595 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.990729 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.990735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:09.990896 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:09.990920 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 22:12:09.991029 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 22:12:09.991231 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1051,"mutex_wait_us":53,"thread_start_us":840,"threads_started":1}
I20260927 22:12:09.992123 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:09.984790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35163 (local address 127.0.97.2:46467)
0927 22:12:09.985042 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:09.985051 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:09.985876 (+ 825us) server_negotiation.cc:407] Connection header received
0927 22:12:09.986114 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:09.986119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:09.986194 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:09.986284 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:12:09.987206 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.988128 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.990043 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.990967 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.991137 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:09.991245 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:09.991553 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 22:12:09.991829 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:12:09.992229 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:09.984775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46467 (local address 127.0.97.1:35163)
0927 22:12:09.985725 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:09.985753 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:09.985935 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:09.986422 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:09.986434 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:09.986530 (+ 96us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:12:09.987039 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:09.987049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.988295 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.988301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:09.989865 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:09.989877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:09.991209 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:09.991215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:09.991388 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:09.991413 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 22:12:09.991526 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 22:12:09.991664 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":1,"spinlock_wait_cycles":43392,"thread_start_us":591,"threads_started":1}
I20260927 22:12:09.993275 15075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08291b00b0874cbfb55389b50d37ab24" candidate_uuid: "ab475a17dfe1486b827647383e529d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" is_pre_election: true
I20260927 22:12:09.993273 15206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08291b00b0874cbfb55389b50d37ab24" candidate_uuid: "ab475a17dfe1486b827647383e529d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730c45f20ad347149922d6f847079c7b" is_pre_election: true
I20260927 22:12:09.993875 15206 raft_consensus.cc:2454] T 08291b00b0874cbfb55389b50d37ab24 P 730c45f20ad347149922d6f847079c7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab475a17dfe1486b827647383e529d22 in term 0.
I20260927 22:12:09.993867 15075 raft_consensus.cc:2454] T 08291b00b0874cbfb55389b50d37ab24 P ccb5cf86772c48a5a859c0d18cdc200f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab475a17dfe1486b827647383e529d22 in term 0.
I20260927 22:12:09.994724 14879 leader_election.cc:302] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 730c45f20ad347149922d6f847079c7b, ab475a17dfe1486b827647383e529d22, ccb5cf86772c48a5a859c0d18cdc200f; no voters:
I20260927 22:12:09.995162 15274 raft_consensus.cc:2781] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:12:09.995313 15274 raft_consensus.cc:482] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:12:09.995450 15274 raft_consensus.cc:3037] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:12:09.998230 15274 raft_consensus.cc:504] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } } peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } }
I20260927 22:12:09.999022 15274 leader_election.cc:288] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 1 election: Requested vote from peers 730c45f20ad347149922d6f847079c7b (127.0.97.3:36819), ccb5cf86772c48a5a859c0d18cdc200f (127.0.97.2:46467)
I20260927 22:12:09.999579 15075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08291b00b0874cbfb55389b50d37ab24" candidate_uuid: "ab475a17dfe1486b827647383e529d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb5cf86772c48a5a859c0d18cdc200f"
I20260927 22:12:09.999671 15206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08291b00b0874cbfb55389b50d37ab24" candidate_uuid: "ab475a17dfe1486b827647383e529d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730c45f20ad347149922d6f847079c7b"
I20260927 22:12:09.999797 15075 raft_consensus.cc:3037] T 08291b00b0874cbfb55389b50d37ab24 P ccb5cf86772c48a5a859c0d18cdc200f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:12:09.999941 15206 raft_consensus.cc:3037] T 08291b00b0874cbfb55389b50d37ab24 P 730c45f20ad347149922d6f847079c7b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:12:10.002125 15075 raft_consensus.cc:2454] T 08291b00b0874cbfb55389b50d37ab24 P ccb5cf86772c48a5a859c0d18cdc200f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab475a17dfe1486b827647383e529d22 in term 1.
I20260927 22:12:10.002398 15206 raft_consensus.cc:2454] T 08291b00b0874cbfb55389b50d37ab24 P 730c45f20ad347149922d6f847079c7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab475a17dfe1486b827647383e529d22 in term 1.
I20260927 22:12:10.002628 14876 leader_election.cc:302] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab475a17dfe1486b827647383e529d22, ccb5cf86772c48a5a859c0d18cdc200f; no voters:
I20260927 22:12:10.003002 15274 raft_consensus.cc:2781] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:12:10.003515 15274 raft_consensus.cc:686] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 1 LEADER]: Becoming Leader. State: Replica: ab475a17dfe1486b827647383e529d22, State: Running, Role: LEADER
I20260927 22:12:10.004163 15274 consensus_queue.cc:237] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } } peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } }
I20260927 22:12:10.009105 14801 catalog_manager.cc:5495] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab475a17dfe1486b827647383e529d22 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "ab475a17dfe1486b827647383e529d22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } health_report { overall_health: UNKNOWN } } }
I20260927 22:12:10.054774 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:10.048898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.0.97.1:40449)
0927 22:12:10.049201 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:10.049209 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:10.049260 (+ 51us) server_negotiation.cc:407] Connection header received
0927 22:12:10.049382 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:10.049387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:10.049490 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:10.049643 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:12:10.050034 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:10.050848 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:10.052444 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:10.052869 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:10.053017 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:10.053564 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:12:10.054092 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:12:10.054258 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:10.054391 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 22:12:10.054478 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260927 22:12:10.063509 15206 raft_consensus.cc:1261] T 08291b00b0874cbfb55389b50d37ab24 P 730c45f20ad347149922d6f847079c7b [term 1 FOLLOWER]: Refusing update from remote peer ab475a17dfe1486b827647383e529d22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:12:10.063539 15075 raft_consensus.cc:1261] T 08291b00b0874cbfb55389b50d37ab24 P ccb5cf86772c48a5a859c0d18cdc200f [term 1 FOLLOWER]: Refusing update from remote peer ab475a17dfe1486b827647383e529d22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:12:10.064689 15278 consensus_queue.cc:1035] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:12:10.065112 15274 consensus_queue.cc:1035] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:12:10.065305 15252 tablet.cc:2369] T 08291b00b0874cbfb55389b50d37ab24 P 730c45f20ad347149922d6f847079c7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:12:10.086408 15282 mvcc.cc:205] Tried to move back new op lower bound from 7334081044729696256 to 7334081044502020096. Current Snapshot: MvccSnapshot[applied={T|T < 7334081044729696256}]
I20260927 22:12:10.099622 14802 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:42226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:12:10.099843 14802 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:42226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:12:10.103850 14802 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db: Sending DeleteTablet for 3 replicas of tablet 08291b00b0874cbfb55389b50d37ab24
I20260927 22:12:10.105275 15055 tablet_service.cc:1514] Processing DeleteTablet for tablet 08291b00b0874cbfb55389b50d37ab24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:12:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59530
I20260927 22:12:10.105314 14923 tablet_service.cc:1514] Processing DeleteTablet for tablet 08291b00b0874cbfb55389b50d37ab24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:12:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38232
I20260927 22:12:10.105595 15186 tablet_service.cc:1514] Processing DeleteTablet for tablet 08291b00b0874cbfb55389b50d37ab24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:12:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56328
I20260927 22:12:10.106175 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 14839
I20260927 22:12:10.116333 15293 tablet_replica.cc:330] T 08291b00b0874cbfb55389b50d37ab24 P 730c45f20ad347149922d6f847079c7b: stopping tablet replica
I20260927 22:12:10.116683 15291 tablet_replica.cc:330] T 08291b00b0874cbfb55389b50d37ab24 P ccb5cf86772c48a5a859c0d18cdc200f: stopping tablet replica
I20260927 22:12:10.116940 15293 raft_consensus.cc:2229] T 08291b00b0874cbfb55389b50d37ab24 P 730c45f20ad347149922d6f847079c7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:12:10.117344 15291 raft_consensus.cc:2229] T 08291b00b0874cbfb55389b50d37ab24 P ccb5cf86772c48a5a859c0d18cdc200f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:12:10.117558 15293 raft_consensus.cc:2258] T 08291b00b0874cbfb55389b50d37ab24 P 730c45f20ad347149922d6f847079c7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:12:10.118083 15291 raft_consensus.cc:2258] T 08291b00b0874cbfb55389b50d37ab24 P ccb5cf86772c48a5a859c0d18cdc200f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:12:10.119371 15292 tablet_replica.cc:330] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22: stopping tablet replica
I20260927 22:12:10.121193 15291 ts_tablet_manager.cc:1905] T 08291b00b0874cbfb55389b50d37ab24 P ccb5cf86772c48a5a859c0d18cdc200f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:12:10.121193 15293 ts_tablet_manager.cc:1905] T 08291b00b0874cbfb55389b50d37ab24 P 730c45f20ad347149922d6f847079c7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:12:10.123715 15292 raft_consensus.cc:2229] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:12:10.125093 15292 raft_consensus.cc:2258] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:12:10.129050 15291 ts_tablet_manager.cc:1918] T 08291b00b0874cbfb55389b50d37ab24 P ccb5cf86772c48a5a859c0d18cdc200f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:12:10.129242 15291 log.cc:1196] T 08291b00b0874cbfb55389b50d37ab24 P ccb5cf86772c48a5a859c0d18cdc200f: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/08291b00b0874cbfb55389b50d37ab24
I20260927 22:12:10.129570 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:10.107054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38254 (local address 127.0.97.1:40449)
0927 22:12:10.107587 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:10.107593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:10.107610 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:12:10.107664 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:10.107669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:10.107760 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:10.107866 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:10.110684 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:10.111969 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:10.114782 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:10.115667 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:10.115819 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:10.120199 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:10.120217 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:10.120236 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:10.120287 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:10.124706 (+ 4419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:10.125911 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:10.125927 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:10.125931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:10.126194 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:10.126727 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:10.126732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:10.126735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:10.127201 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:10.127554 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:10.128802 (+ 1248us) server_negotiation.cc:299] Negotiation successful
0927 22:12:10.129084 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404}
I20260927 22:12:10.129914 15291 ts_tablet_manager.cc:1939] T 08291b00b0874cbfb55389b50d37ab24 P ccb5cf86772c48a5a859c0d18cdc200f: Deleting consensus metadata
I20260927 22:12:10.130294 14983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:12:10.132055 15293 ts_tablet_manager.cc:1918] T 08291b00b0874cbfb55389b50d37ab24 P 730c45f20ad347149922d6f847079c7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:12:10.132043 14786 catalog_manager.cc:4841] TS ccb5cf86772c48a5a859c0d18cdc200f (127.0.97.2:46467): tablet 08291b00b0874cbfb55389b50d37ab24 (table test-table [id=8c053ea753434fc2875bfd82e6a1e11c]) successfully deleted
I20260927 22:12:10.132265 15293 log.cc:1196] T 08291b00b0874cbfb55389b50d37ab24 P 730c45f20ad347149922d6f847079c7b: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/08291b00b0874cbfb55389b50d37ab24
I20260927 22:12:10.132829 15293 ts_tablet_manager.cc:1939] T 08291b00b0874cbfb55389b50d37ab24 P 730c45f20ad347149922d6f847079c7b: Deleting consensus metadata
I20260927 22:12:10.134538 14789 catalog_manager.cc:4841] TS 730c45f20ad347149922d6f847079c7b (127.0.97.3:36819): tablet 08291b00b0874cbfb55389b50d37ab24 (table test-table [id=8c053ea753434fc2875bfd82e6a1e11c]) successfully deleted
I20260927 22:12:10.232762 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 14839
I20260927 22:12:10.232861 15292 ts_tablet_manager.cc:1905] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 22:12:10.252115 14789 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:40449 (error 108)
I20260927 22:12:10.252429 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 14992
I20260927 22:12:10.254179 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:10.253643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40449 (local address 127.0.0.1:38264)
0927 22:12:10.253841 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:10.253995 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 22:12:10.254467 14789 catalog_manager.cc:4584] TS ab475a17dfe1486b827647383e529d22 (127.0.97.1:40449): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 08291b00b0874cbfb55389b50d37ab24: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40449: connect: Connection refused (error 111)
I20260927 22:12:10.267906 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:10.253355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59534 (local address 127.0.97.2:46467)
0927 22:12:10.253593 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:10.253597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:10.253617 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:12:10.253770 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:10.253778 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:10.253860 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:10.253978 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:10.255531 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:10.256334 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:10.257730 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:10.258144 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:10.258241 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:10.261328 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:10.261343 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:10.261361 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:10.261462 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:10.264825 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:10.265663 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:10.265670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:10.265673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:10.265808 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:10.266232 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:10.266236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:10.266239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:10.266785 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:10.267073 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:10.267476 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 22:12:10.267648 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260927 22:12:10.268662 15115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:12:10.308435 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:10.307867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40449 (local address 127.0.0.1:38278)
0927 22:12:10.308103 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:10.308195 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 22:12:10.309435 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:10.309088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40449 (local address 127.0.0.1:38288)
0927 22:12:10.309236 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:10.309317 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 22:12:10.358623 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 14992
I20260927 22:12:10.370363 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:10.369769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40449 (local address 127.0.0.1:38296)
0927 22:12:10.370016 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:10.370136 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 22:12:10.371521 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:10.371083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40449 (local address 127.0.0.1:38300)
0927 22:12:10.371263 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:10.371359 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 22:12:10.377949 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 15123
I20260927 22:12:10.391924 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:10.378755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56330 (local address 127.0.97.3:36819)
0927 22:12:10.378976 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:10.378982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:10.379003 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:12:10.379175 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:10.379185 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:10.379251 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:10.379358 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:10.380749 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:10.381478 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:10.382882 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:10.383297 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:10.383428 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:10.386053 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:10.386068 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:10.386086 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:10.386134 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:10.389036 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:10.389808 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:10.389824 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:10.389828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:10.389982 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:10.390374 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:10.390379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:10.390388 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:10.390847 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:10.391148 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:10.391487 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 22:12:10.391667 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:12:10.392685 15246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:12:10.452172 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:10.451550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40449 (local address 127.0.0.1:38308)
0927 22:12:10.451831 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:10.451977 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260927 22:12:10.453418 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:10.452897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40449 (local address 127.0.0.1:38316)
0927 22:12:10.453135 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:10.453230 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260927 22:12:10.485152 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 15123
I20260927 22:12:10.502344 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 14769
I20260927 22:12:10.503650 14832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:12:10.619426 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 14769
I20260927 22:12:10.641973 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42769
--webserver_interface=127.0.97.62
--webserver_port=40201
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42769
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:12:10.977591 15299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:12:10.977933 15299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:12:10.978130 15299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:12:10.985409 15299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:12:10.985555 15299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:12:10.985682 15299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:12:10.985780 15299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:12:10.985865 15299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:12:10.995996 15299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=40201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:12:10.997475 15299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:12:10.998894 15299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:12:11.010895 15304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:11.011147 15305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:11.011308 15307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:12:11.012086 15299 server_base.cc:1034] running on GCE node
I20260927 22:12:11.012960 15299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:12:11.014467 15299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:12:11.015703 15299 hybrid_clock.cc:648] HybridClock initialized: now 1790547131015639 us; error 42 us; skew 500 ppm
I20260927 22:12:11.016103 15299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:12:11 dist-test-slave-mp03 krb5kdc[14215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:12:11.019372 15299 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:12:11.020859 15299 webserver.cc:460] Webserver started at http://127.0.97.62:40201/ using document root <none> and password file <none>
I20260927 22:12:11.021589 15299 fs_manager.cc:359] Metadata directory not provided
I20260927 22:12:11.021740 15299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:12:11.027140 15299 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260927 22:12:11.029870 15314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:11.031281 15299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:12:11.031510 15299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "6bf417a069e240edac650dfc6da196db"
format_stamp: "Formatted at 2026-09-27 22:12:05 on dist-test-slave-mp03"
server_key: "a8a3c928021a68c72936fa0c2a3d947d"
server_key_iv: "a20943fca365a6292a0405f1bb0bbb57"
server_key_version: "encryptionkey@0"
I20260927 22:12:11.032516 15299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:12:11.046130 15299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:12:11.050197 15299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:12:11.050480 15299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:12:11.069099 15299 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42769
I20260927 22:12:11.069123 15365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42769 every 8 connection(s)
I20260927 22:12:11.070624 15299 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:12:11.075608 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 15299
I20260927 22:12:11.076761 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40449
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42769
--webserver_port=45427
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:12:11.079699 15366 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:12:11.084614 15366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db: Bootstrap starting.
I20260927 22:12:11.104844 15366 log.cc:824] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db: Log is configured to *not* fsync() on all Append() calls
I20260927 22:12:11.120960 15366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:12:11.121553 15366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db: Bootstrap complete.
I20260927 22:12:11.131147 15366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf417a069e240edac650dfc6da196db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42769 } }
I20260927 22:12:11.132104 15366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bf417a069e240edac650dfc6da196db, State: Initialized, Role: FOLLOWER
I20260927 22:12:11.132736 15366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf417a069e240edac650dfc6da196db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42769 } }
I20260927 22:12:11.132983 15366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:12:11.133114 15366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:12:11.133288 15366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [term 2 FOLLOWER]: Advancing to term 3
I20260927 22:12:11.137499 15366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf417a069e240edac650dfc6da196db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42769 } }
I20260927 22:12:11.138041 15366 leader_election.cc:302] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bf417a069e240edac650dfc6da196db; no voters:
I20260927 22:12:11.138767 15366 leader_election.cc:288] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 22:12:11.138906 15370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [term 3 FOLLOWER]: Leader election won for term 3
I20260927 22:12:11.141150 15366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:12:11.146039 15370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [term 3 LEADER]: Becoming Leader. State: Replica: 6bf417a069e240edac650dfc6da196db, State: Running, Role: LEADER
I20260927 22:12:11.146741 15370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf417a069e240edac650dfc6da196db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42769 } }
I20260927 22:12:11.149665 15371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6bf417a069e240edac650dfc6da196db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf417a069e240edac650dfc6da196db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42769 } } }
I20260927 22:12:11.150101 15371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [sys.catalog]: This master's current role is: LEADER
I20260927 22:12:11.150758 15372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bf417a069e240edac650dfc6da196db. Latest consensus state: current_term: 3 leader_uuid: "6bf417a069e240edac650dfc6da196db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf417a069e240edac650dfc6da196db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42769 } } }
I20260927 22:12:11.151079 15372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf417a069e240edac650dfc6da196db [sys.catalog]: This master's current role is: LEADER
I20260927 22:12:11.159644 15385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:12:11.166568 15385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:12:11.167701 15385 catalog_manager.cc:1259] Loaded cluster ID: 559aee8a582e415f9d4d686f910098f6
I20260927 22:12:11.167811 15385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 22:12:11.169231 15385 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2393c879ad google::LogMessage::Fail() at ??:0
@ 0x7f2393c8a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f2393c874de google::LogMessage::Flush() at ??:0
@ 0x7f2393c88a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f23a96be512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f23a96bb976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f23a9743610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f23a9743439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f23968d3345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f23968d691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f23968d6799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f23968b1eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2399b8b6db start_thread at ??:0
@ 0x7f2392b2161f clone at ??:0
W20260927 22:12:11.455171 15368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:12:11.455535 15368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:12:11.455722 15368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:12:11.463281 15368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:12:11.463434 15368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:12:11.463613 15368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:12:11.473802 15368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45427
--tserver_master_addrs=127.0.97.62:42769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:12:11.475324 15368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:12:11.476783 15368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:12:11.491060 15408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:11.491102 15409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:11.490993 15411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:12:11.492085 15368 server_base.cc:1034] running on GCE node
I20260927 22:12:11.492719 15368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:12:11.493705 15368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:12:11.494937 15368 hybrid_clock.cc:648] HybridClock initialized: now 1790547131494894 us; error 59 us; skew 500 ppm
I20260927 22:12:11.495277 15368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:12:11 dist-test-slave-mp03 krb5kdc[14215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:12:11.498492 15368 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:12:11.500125 15368 webserver.cc:460] Webserver started at http://127.0.97.1:45427/ using document root <none> and password file <none>
I20260927 22:12:11.500831 15368 fs_manager.cc:359] Metadata directory not provided
I20260927 22:12:11.500993 15368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:12:11.506479 15368 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:12:11.511619 15418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:11.512966 15368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:12:11.513170 15368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "ab475a17dfe1486b827647383e529d22"
format_stamp: "Formatted at 2026-09-27 22:12:05 on dist-test-slave-mp03"
server_key: "d8ac84e7dbdcb0a2cdfe30346199521c"
server_key_iv: "75cbc51e1d542fa1786ccea638b27137"
server_key_version: "encryptionkey@0"
I20260927 22:12:11.514200 15368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:12:11.552286 15368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:12:11.557528 15368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:12:11.557865 15368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:12:11.559059 15368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:12:11.560868 15425 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260927 22:12:11.567255 15368 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 22:12:11.567401 15368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260927 22:12:11.567550 15368 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 22:12:11.570755 15368 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 22:12:11.570855 15368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260927 22:12:11.571179 15425 tablet_bootstrap.cc:492] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22: Bootstrap starting.
I20260927 22:12:11.579924 15425 log.cc:824] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22: Log is configured to *not* fsync() on all Append() calls
I20260927 22:12:11.596338 15425 tablet_bootstrap.cc:492] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:12:11.597052 15425 tablet_bootstrap.cc:492] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22: Bootstrap complete.
I20260927 22:12:11.597647 15425 ts_tablet_manager.cc:1397] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22: Time spent bootstrapping tablet: real 0.027s user 0.019s sys 0.004s
I20260927 22:12:11.608819 15425 raft_consensus.cc:348] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } } peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } }
I20260927 22:12:11.610212 15425 raft_consensus.cc:729] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab475a17dfe1486b827647383e529d22, State: Initialized, Role: FOLLOWER
I20260927 22:12:11.611027 15425 consensus_queue.cc:260] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } } peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } }
I20260927 22:12:11.612150 15368 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40449
I20260927 22:12:11.612190 15533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40449 every 8 connection(s)
I20260927 22:12:11.612947 15425 ts_tablet_manager.cc:1428] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22: Time spent starting tablet: real 0.015s user 0.017s sys 0.000s
I20260927 22:12:11.613505 15368 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:12:11.618038 15536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:11.616130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.1:60215)
0927 22:12:11.616920 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:11.617100 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":365,"threads_started":1}
W20260927 22:12:11.618752 15534 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42769 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:42769: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
I20260927 22:12:11.619549 15536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:11.619232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.1:35805)
0927 22:12:11.619359 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:11.619425 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 22:12:11.620374 15536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:11.620047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.1:54345)
0927 22:12:11.620190 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:11.620267 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 22:12:11.620637 15534 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:12:11.623806 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 15368
I20260927 22:12:11.624415 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:46467
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42769
--webserver_port=37281
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:12:11.964697 15537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:12:11.965032 15537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:12:11.965193 15537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:12:11.972707 15537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:12:11.972862 15537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:12:11.973052 15537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:12:11.982856 15537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:46467
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=37281
--tserver_master_addrs=127.0.97.62:42769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:12:11.984238 15537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:12:11.985620 15537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:12:11.999491 15542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:11.999459 15543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:11.999513 15545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:12:12.000347 15537 server_base.cc:1034] running on GCE node
I20260927 22:12:12.000973 15537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:12:12.001940 15537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:12:12.003254 15537 hybrid_clock.cc:648] HybridClock initialized: now 1790547132003151 us; error 47 us; skew 500 ppm
I20260927 22:12:12.003569 15537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:12:12 dist-test-slave-mp03 krb5kdc[14215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:12:12.012223 15537 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:12:12.014029 15537 webserver.cc:460] Webserver started at http://127.0.97.2:37281/ using document root <none> and password file <none>
I20260927 22:12:12.014868 15537 fs_manager.cc:359] Metadata directory not provided
I20260927 22:12:12.015017 15537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:12:12.020742 15537 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:12:12.023854 15552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:12.025491 15537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:12:12.025681 15537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "ccb5cf86772c48a5a859c0d18cdc200f"
format_stamp: "Formatted at 2026-09-27 22:12:06 on dist-test-slave-mp03"
server_key: "ce362b06f1d75dcc2a4ad2307ab553a8"
server_key_iv: "d57ac84eab760d1d20a5a14d18c7d209"
server_key_version: "encryptionkey@0"
I20260927 22:12:12.026789 15537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:12:12.040023 15537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:12:12.044574 15537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:12:12.044868 15537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:12:12.045853 15537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:12:12.047272 15537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:12:12.047405 15537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:12.047534 15537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:12:12.047623 15537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:12.080343 15537 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46467
I20260927 22:12:12.080435 15664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46467 every 8 connection(s)
I20260927 22:12:12.081854 15537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:12:12.087137 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:12.085065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.2:42523)
0927 22:12:12.085936 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:12.086153 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":320,"threads_started":1}
W20260927 22:12:12.087698 15665 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42769 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:42769: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
I20260927 22:12:12.088572 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:12.088215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.2:41149)
0927 22:12:12.088358 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:12.088461 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 22:12:12.089497 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:12.089168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.2:60743)
0927 22:12:12.089290 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:12.089361 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 22:12:12.089767 15665 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:12:12.090288 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 15537
I20260927 22:12:12.091034 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36819
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42769
--webserver_port=35283
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:12:12.422228 15668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:12:12.422551 15668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:12:12.422751 15668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:12:12.430455 15668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:12:12.430629 15668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:12:12.430783 15668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:12:12.441447 15668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35283
--tserver_master_addrs=127.0.97.62:42769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:12:12.442795 15668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:12:12.444098 15668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:12:12.457494 15673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:12.457579 15674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:12.457530 15676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:12:12.463758 15668 server_base.cc:1034] running on GCE node
I20260927 22:12:12.464484 15668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:12:12.465530 15668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:12:12.466776 15668 hybrid_clock.cc:648] HybridClock initialized: now 1790547132466733 us; error 45 us; skew 500 ppm
I20260927 22:12:12.467121 15668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:12:12 dist-test-slave-mp03 krb5kdc[14215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:12:12.470561 15668 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:12:12.472280 15668 webserver.cc:460] Webserver started at http://127.0.97.3:35283/ using document root <none> and password file <none>
I20260927 22:12:12.473083 15668 fs_manager.cc:359] Metadata directory not provided
I20260927 22:12:12.473266 15668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:12:12.479700 15668 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 22:12:12.482645 15683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:12.484220 15668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:12:12.484381 15668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "730c45f20ad347149922d6f847079c7b"
format_stamp: "Formatted at 2026-09-27 22:12:06 on dist-test-slave-mp03"
server_key: "9296e21010cfbf7e074a3d8fbeb9be8f"
server_key_iv: "de87720459ad3ca945dc44a835fec3dd"
server_key_version: "encryptionkey@0"
I20260927 22:12:12.485356 15668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:12:12.498862 15668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:12:12.503338 15668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:12:12.503575 15668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:12:12.504582 15668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:12:12.505817 15668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:12:12.505904 15668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:12.506045 15668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:12:12.506153 15668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:12.539670 15668 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36819
I20260927 22:12:12.539757 15795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36819 every 8 connection(s)
I20260927 22:12:12.541178 15668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:12:12.544943 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 15668
I20260927 22:12:12.546644 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:12.544272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.3:54733)
0927 22:12:12.545181 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:12.545393 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":350,"threads_started":1}
W20260927 22:12:12.547320 15796 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42769 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:42769: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
I20260927 22:12:12.548079 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:12.547787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.3:51517)
0927 22:12:12.547915 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:12.547979 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 22:12:12.548847 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:12.548575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.3:34783)
0927 22:12:12.548684 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:12.548751 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 22:12:12.549088 15796 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:12:12.621840 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:12.621120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.1:52911)
0927 22:12:12.621541 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:12.621635 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":197,"threads_started":1}
I20260927 22:12:13.091480 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:13.090311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.2:34501)
0927 22:12:13.090858 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:13.091042 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":236,"threads_started":1}
I20260927 22:12:13.160676 15802 raft_consensus.cc:482] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:12:13.160945 15802 raft_consensus.cc:504] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } } peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } }
I20260927 22:12:13.162462 15802 leader_election.cc:288] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 730c45f20ad347149922d6f847079c7b (127.0.97.3:36819), ccb5cf86772c48a5a859c0d18cdc200f (127.0.97.2:46467)
Sep 27 22:12:13 dist-test-slave-mp03 krb5kdc[14215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
Sep 27 22:12:13 dist-test-slave-mp03 krb5kdc[14215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:12:13.200537 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:13.163190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41247 (local address 127.0.97.2:46467)
0927 22:12:13.178316 (+ 15126us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:13.178330 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:13.178383 (+ 53us) server_negotiation.cc:407] Connection header received
0927 22:12:13.180627 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:13.180634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:13.181047 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:13.181425 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:13.183665 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:13.184892 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:13.186626 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:13.187318 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:13.187465 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:13.192282 (+ 4817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:13.192290 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:13.192305 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:13.192383 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:13.196514 (+ 4131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:13.197388 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:13.197397 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:13.197407 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:13.197628 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:13.198092 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:13.198102 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:13.198105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:13.198772 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:13.198973 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:13.199749 (+ 776us) server_negotiation.cc:299] Negotiation successful
0927 22:12:13.200133 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14900,"thread_start_us":14694,"threads_started":1}
I20260927 22:12:13.200562 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:13.163120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46197 (local address 127.0.97.3:36819)
0927 22:12:13.177682 (+ 14562us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:13.177709 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:13.178676 (+ 967us) server_negotiation.cc:407] Connection header received
0927 22:12:13.183659 (+ 4983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:13.183667 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:13.184018 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:13.184346 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:13.185452 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:13.186598 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:13.188048 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:13.188972 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:13.189108 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:13.192755 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:13.192767 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:13.192782 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:13.192825 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:13.197057 (+ 4232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:13.197856 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:13.197864 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:13.197871 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:13.198049 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:13.198437 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:13.198450 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:13.198454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:13.199001 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:13.199186 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:13.199749 (+ 563us) server_negotiation.cc:299] Negotiation successful
0927 22:12:13.200106 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14286,"thread_start_us":14158,"threads_started":1}
I20260927 22:12:13.200585 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:13.163022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36819 (local address 127.0.97.1:46197)
0927 22:12:13.178532 (+ 15510us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:13.178549 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:13.178825 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:13.184301 (+ 5476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:13.184306 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:13.184900 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:12:13.185276 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:13.185293 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:13.186761 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:13.186765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:13.187907 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:13.187920 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:13.189162 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:13.189171 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:13.189389 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:13.190077 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:12:13.190104 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:12:13.192469 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:12:13.197213 (+ 4744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:13.197219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:13.197223 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:13.197736 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:13.198179 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:13.198184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:13.198188 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:13.198323 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:13.199140 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:12:13.199145 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:12:13.199326 (+ 181us) client_negotiation.cc:769] Sending connection context
0927 22:12:13.199524 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 22:12:13.200106 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15391,"mutex_wait_us":60,"thread_start_us":15301,"threads_started":1}
I20260927 22:12:13.200752 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:13.163022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46467 (local address 127.0.97.1:41247)
0927 22:12:13.178075 (+ 15053us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:13.178135 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:13.178852 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:13.181677 (+ 2825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:13.181690 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:13.182573 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:12:13.183450 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:13.183479 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:13.185121 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:13.185128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:13.186460 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:13.186472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:13.187508 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:13.187515 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:13.187832 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:13.188838 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:12:13.188862 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:12:13.191960 (+ 3098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:12:13.196713 (+ 4753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:13.196721 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:13.196734 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:13.197222 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:13.197749 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:13.197752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:13.197754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:13.197937 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:13.198923 (+ 986us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:12:13.198932 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:12:13.199257 (+ 325us) client_negotiation.cc:769] Sending connection context
0927 22:12:13.199526 (+ 269us) client_negotiation.cc:240] Negotiation successful
0927 22:12:13.200271 (+ 745us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14914,"mutex_wait_us":282,"spinlock_wait_cycles":40320,"thread_start_us":15200,"threads_started":1}
W20260927 22:12:13.204403 15422 leader_election.cc:341] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 730c45f20ad347149922d6f847079c7b (127.0.97.3:36819): Not found: Tablet not found: 08291b00b0874cbfb55389b50d37ab24
W20260927 22:12:13.204701 15419 leader_election.cc:341] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ccb5cf86772c48a5a859c0d18cdc200f (127.0.97.2:46467): Not found: Tablet not found: 08291b00b0874cbfb55389b50d37ab24
I20260927 22:12:13.204865 15419 leader_election.cc:302] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab475a17dfe1486b827647383e529d22; no voters: 730c45f20ad347149922d6f847079c7b, ccb5cf86772c48a5a859c0d18cdc200f
I20260927 22:12:13.205287 15802 raft_consensus.cc:2726] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 22:12:13.550572 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:13.549694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.3:43941)
0927 22:12:13.550164 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:13.550304 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":229,"threads_started":1}
I20260927 22:12:13.623211 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:13.622783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.1:40963)
0927 22:12:13.622972 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:13.623067 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 22:12:14.093256 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:14.092333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.2:37593)
0927 22:12:14.092976 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:14.093078 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":411,"threads_started":1}
I20260927 22:12:14.552189 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:14.551533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.3:44229)
0927 22:12:14.551957 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:14.552035 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":212,"threads_started":1}
I20260927 22:12:14.625083 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:14.624172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.1:39425)
0927 22:12:14.624744 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:14.624849 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":306,"threads_started":1}
I20260927 22:12:14.875357 15811 raft_consensus.cc:482] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:12:14.875567 15811 raft_consensus.cc:504] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } } peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } }
I20260927 22:12:14.876196 15811 leader_election.cc:288] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 730c45f20ad347149922d6f847079c7b (127.0.97.3:36819), ccb5cf86772c48a5a859c0d18cdc200f (127.0.97.2:46467)
W20260927 22:12:14.877588 15422 leader_election.cc:341] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 730c45f20ad347149922d6f847079c7b (127.0.97.3:36819): Not found: Tablet not found: 08291b00b0874cbfb55389b50d37ab24
W20260927 22:12:14.877841 15419 leader_election.cc:341] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ccb5cf86772c48a5a859c0d18cdc200f (127.0.97.2:46467): Not found: Tablet not found: 08291b00b0874cbfb55389b50d37ab24
I20260927 22:12:14.878014 15419 leader_election.cc:302] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab475a17dfe1486b827647383e529d22; no voters: 730c45f20ad347149922d6f847079c7b, ccb5cf86772c48a5a859c0d18cdc200f
I20260927 22:12:14.878340 15811 raft_consensus.cc:2726] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 22:12:15.095182 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:15.094271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.2:50945)
0927 22:12:15.094912 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:15.095012 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":380,"threads_started":1}
I20260927 22:12:15.554008 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:15.553255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.3:53213)
0927 22:12:15.553757 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:15.553841 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":277,"threads_started":1}
I20260927 22:12:15.626641 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:15.625970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.1:36167)
0927 22:12:15.626379 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:15.626455 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":220,"threads_started":1}
I20260927 22:12:16.096940 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:16.096097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.2:33091)
0927 22:12:16.096671 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:16.096756 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":265,"threads_started":1}
I20260927 22:12:16.555805 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:16.555038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.3:35117)
0927 22:12:16.555528 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:16.555621 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":221,"threads_started":1}
I20260927 22:12:16.628768 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:16.627842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.1:36947)
0927 22:12:16.628439 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:16.628562 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":318,"threads_started":1}
I20260927 22:12:17.098662 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:17.097882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.2:50989)
0927 22:12:17.098343 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:17.098427 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":237,"threads_started":1}
I20260927 22:12:17.347682 15819 raft_consensus.cc:482] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:12:17.347930 15819 raft_consensus.cc:504] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } } peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } }
I20260927 22:12:17.348739 15819 leader_election.cc:288] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 730c45f20ad347149922d6f847079c7b (127.0.97.3:36819), ccb5cf86772c48a5a859c0d18cdc200f (127.0.97.2:46467)
W20260927 22:12:17.349736 15422 leader_election.cc:341] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 730c45f20ad347149922d6f847079c7b (127.0.97.3:36819): Not found: Tablet not found: 08291b00b0874cbfb55389b50d37ab24
W20260927 22:12:17.349936 15419 leader_election.cc:341] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ccb5cf86772c48a5a859c0d18cdc200f (127.0.97.2:46467): Not found: Tablet not found: 08291b00b0874cbfb55389b50d37ab24
I20260927 22:12:17.350076 15419 leader_election.cc:302] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab475a17dfe1486b827647383e529d22; no voters: 730c45f20ad347149922d6f847079c7b, ccb5cf86772c48a5a859c0d18cdc200f
I20260927 22:12:17.350381 15819 raft_consensus.cc:2726] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 22:12:17.557569 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:17.556735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.3:60355)
0927 22:12:17.557232 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:17.557380 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":275,"threads_started":1}
I20260927 22:12:17.630887 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:17.629916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.1:54753)
0927 22:12:17.630512 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:17.630664 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":327,"threads_started":1}
I20260927 22:12:18.100304 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:18.099503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.2:57819)
0927 22:12:18.100029 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:18.100111 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":308,"threads_started":1}
I20260927 22:12:18.559516 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:18.558554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.3:51403)
0927 22:12:18.559245 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:18.559327 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":262,"threads_started":1}
I20260927 22:12:18.632545 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:18.631818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.1:46415)
0927 22:12:18.632262 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:18.632349 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":231,"threads_started":1}
I20260927 22:12:18.900138 15825 raft_consensus.cc:482] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:12:18.900389 15825 raft_consensus.cc:504] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } } peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } }
I20260927 22:12:18.901243 15825 leader_election.cc:288] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 730c45f20ad347149922d6f847079c7b (127.0.97.3:36819), ccb5cf86772c48a5a859c0d18cdc200f (127.0.97.2:46467)
W20260927 22:12:18.902297 15422 leader_election.cc:341] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 730c45f20ad347149922d6f847079c7b (127.0.97.3:36819): Not found: Tablet not found: 08291b00b0874cbfb55389b50d37ab24
W20260927 22:12:18.902506 15419 leader_election.cc:341] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ccb5cf86772c48a5a859c0d18cdc200f (127.0.97.2:46467): Not found: Tablet not found: 08291b00b0874cbfb55389b50d37ab24
I20260927 22:12:18.902658 15419 leader_election.cc:302] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab475a17dfe1486b827647383e529d22; no voters: 730c45f20ad347149922d6f847079c7b, ccb5cf86772c48a5a859c0d18cdc200f
I20260927 22:12:18.902994 15825 raft_consensus.cc:2726] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 22:12:19.102159 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:19.101230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.2:35911)
0927 22:12:19.101853 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:19.101938 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":311,"threads_started":1}
I20260927 22:12:19.561327 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:19.560458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.3:42503)
0927 22:12:19.561025 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:19.561114 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":298,"threads_started":1}
I20260927 22:12:19.634423 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:19.633663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.1:42817)
0927 22:12:19.634160 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:19.634248 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":271,"threads_started":1}
I20260927 22:12:20.103852 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:20.103180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.2:57759)
0927 22:12:20.103599 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:20.103681 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":199,"threads_started":1}
I20260927 22:12:20.563006 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:20.562243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.3:50923)
0927 22:12:20.562725 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:20.562813 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":215,"threads_started":1}
I20260927 22:12:20.636165 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:20.635422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.1:42223)
0927 22:12:20.635895 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:20.635987 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":231,"threads_started":1}
I20260927 22:12:21.105970 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:21.104945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.2:48993)
0927 22:12:21.105606 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:21.105739 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":307,"threads_started":1}
I20260927 22:12:21.564599 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:21.563884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.3:43657)
0927 22:12:21.564367 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:21.564443 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":243,"threads_started":1}
I20260927 22:12:21.638379 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:21.637356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.1:43413)
0927 22:12:21.637966 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:21.638107 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":281,"threads_started":1}
I20260927 22:12:22.107843 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:22.107056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.2:41551)
0927 22:12:22.107577 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:22.107661 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":300,"threads_started":1}
I20260927 22:12:22.566359 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:22.565576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.3:43217)
0927 22:12:22.566084 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:22.566167 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":304,"threads_started":1}
I20260927 22:12:22.640369 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:22.639412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.1:36739)
0927 22:12:22.640044 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:22.640173 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":368,"threads_started":1}
I20260927 22:12:23.109601 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:23.108813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.2:49369)
0927 22:12:23.109345 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:23.109431 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":277,"threads_started":1}
I20260927 22:12:23.568118 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:23.567355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.3:52081)
0927 22:12:23.567871 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:23.567958 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":218,"threads_started":1}
I20260927 22:12:23.642433 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:23.641505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.1:60721)
0927 22:12:23.642088 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:23.642206 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":284,"threads_started":1}
I20260927 22:12:24.111434 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:24.110562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.2:44077)
0927 22:12:24.111135 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:24.111235 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":272,"threads_started":1}
I20260927 22:12:24.569909 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:24.569191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.3:53439)
0927 22:12:24.569649 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:24.569732 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":240,"threads_started":1}
I20260927 22:12:24.644798 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:24.643888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.1:35301)
0927 22:12:24.644472 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:24.644577 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":311,"threads_started":1}
I20260927 22:12:25.113222 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:25.112446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.2:41067)
0927 22:12:25.112973 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:25.113057 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":295,"threads_started":1}
I20260927 22:12:25.566531 15845 raft_consensus.cc:482] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:12:25.566900 15845 raft_consensus.cc:504] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab475a17dfe1486b827647383e529d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40449 } } peers { permanent_uuid: "730c45f20ad347149922d6f847079c7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36819 } } peers { permanent_uuid: "ccb5cf86772c48a5a859c0d18cdc200f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46467 } }
I20260927 22:12:25.567813 15845 leader_election.cc:288] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 730c45f20ad347149922d6f847079c7b (127.0.97.3:36819), ccb5cf86772c48a5a859c0d18cdc200f (127.0.97.2:46467)
W20260927 22:12:25.568837 15419 leader_election.cc:341] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ccb5cf86772c48a5a859c0d18cdc200f (127.0.97.2:46467): Not found: Tablet not found: 08291b00b0874cbfb55389b50d37ab24
W20260927 22:12:25.569020 15422 leader_election.cc:341] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 730c45f20ad347149922d6f847079c7b (127.0.97.3:36819): Not found: Tablet not found: 08291b00b0874cbfb55389b50d37ab24
I20260927 22:12:25.569182 15422 leader_election.cc:302] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab475a17dfe1486b827647383e529d22; no voters: 730c45f20ad347149922d6f847079c7b, ccb5cf86772c48a5a859c0d18cdc200f
I20260927 22:12:25.569483 15845 raft_consensus.cc:2726] T 08291b00b0874cbfb55389b50d37ab24 P ab475a17dfe1486b827647383e529d22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 22:12:25.571887 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:25.571069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.3:46091)
0927 22:12:25.571610 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:25.571691 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":314,"threads_started":1}
I20260927 22:12:25.646895 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:25.645834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.1:54023)
0927 22:12:25.646388 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:25.646505 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":279,"threads_started":1}
I20260927 22:12:26.115049 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:26.114185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.2:57167)
0927 22:12:26.114707 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:26.114847 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":238,"threads_started":1}
I20260927 22:12:26.573683 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:26.572871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.3:54959)
0927 22:12:26.573380 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:26.573471 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":260,"threads_started":1}
I20260927 22:12:26.648958 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:26.648259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.1:58191)
0927 22:12:26.648692 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:26.648783 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":215,"threads_started":1}
I20260927 22:12:27.117035 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:27.116169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.2:38919)
0927 22:12:27.116668 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:27.116814 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":261,"threads_started":1}
I20260927 22:12:27.550356 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 15368
I20260927 22:12:27.567857 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:27.551776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52850 (local address 127.0.97.1:40449)
0927 22:12:27.552502 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:27.552511 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:27.552544 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:12:27.552629 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:27.552639 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:27.552998 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:27.553193 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:27.554890 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:27.555995 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:27.557274 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:27.557960 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:27.558068 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:27.561393 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:27.561408 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:27.561426 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:27.561466 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:27.564823 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:27.565648 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:27.565654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:27.565656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:27.565830 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:27.566198 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:27.566201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:27.566203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:27.566569 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:27.566903 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:27.567210 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 22:12:27.567423 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":383,"threads_started":1}
I20260927 22:12:27.569186 15528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:12:27.575693 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:27.574895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.3:59755)
0927 22:12:27.575442 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:27.575543 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":299,"threads_started":1}
I20260927 22:12:27.660370 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:27.659218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42769 (local address 127.0.97.1:53269)
0927 22:12:27.659825 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:27.660015 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":293,"threads_started":1}
I20260927 22:12:27.660534 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 15368
I20260927 22:12:27.679024 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 15537
I20260927 22:12:27.693292 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:27.679831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44360 (local address 127.0.97.2:46467)
0927 22:12:27.680331 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:27.680336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:27.680353 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:12:27.680410 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:27.680415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:27.680476 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:27.680566 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:27.681944 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:27.682894 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:27.684156 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:27.684590 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:27.684685 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:27.687444 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:27.687450 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:27.687455 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:27.687492 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:27.690561 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:27.691325 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:27.691329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:27.691332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:27.691457 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:27.691835 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:27.691839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:27.691842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:27.692140 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:27.692368 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:27.692680 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 22:12:27.692859 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":268,"threads_started":1}
I20260927 22:12:27.694068 15660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:12:27.778028 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 15537
I20260927 22:12:27.796084 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 15668
I20260927 22:12:27.810508 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:27.797052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49218 (local address 127.0.97.3:36819)
0927 22:12:27.797624 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:27.797628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:27.797645 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:12:27.797699 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:27.797702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:27.797832 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:27.797943 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:27.799409 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:27.800191 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:27.801508 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:27.801973 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:27.802074 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:27.804888 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:27.804893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:27.804897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:27.804933 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:27.807997 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:27.808630 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:27.808634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:27.808636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:27.808813 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:27.809148 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:27.809151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:27.809153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:27.809428 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:27.809690 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:27.809936 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 22:12:27.810112 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":298,"threads_started":1}
I20260927 22:12:27.811460 15791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:12:27.896242 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 15668
2026-09-27T22:12:27Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24509 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:12:27 dist-test-slave-mp03 krb5kdc[15862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:12:27 dist-test-slave-mp03 krb5kdc[15862](info): set up 2 sockets
Sep 27 22:12:27 dist-test-slave-mp03 krb5kdc[15862](info): commencing operation
krb5kdc: starting...
W20260927 22:12:29.007323 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:12:29 dist-test-slave-mp03 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:12:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:12:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:12:29.262382 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32915
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:12:29.601130 15875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:12:29.601447 15875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:12:29.601537 15875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:12:29.608999 15875 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 22:12:29.609107 15875 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 22:12:29.609144 15875 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 22:12:29.609192 15875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:12:29.609246 15875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:12:29.609313 15875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:12:29.609359 15875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:12:29.609405 15875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:12:29.619606 15875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:12:29.620878 15875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:12:29.622191 15875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:12:29.634639 15883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:29.634675 15881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:29.634786 15880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:12:29.635336 15875 server_base.cc:1034] running on GCE node
I20260927 22:12:29.636207 15875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:12:29.637737 15875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:12:29.639005 15875 hybrid_clock.cc:648] HybridClock initialized: now 1790547149638947 us; error 33 us; skew 500 ppm
I20260927 22:12:29.639380 15875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:12:29 dist-test-slave-mp03 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:12:29.642777 15875 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:12:29.644296 15875 webserver.cc:460] Webserver started at http://127.0.97.62:39915/ using document root <none> and password file <none>
I20260927 22:12:29.645015 15875 fs_manager.cc:359] Metadata directory not provided
I20260927 22:12:29.645099 15875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:12:29.645370 15875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:12:29.648468 15875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "e40f63c65f4f49858110085b873b4028"
format_stamp: "Formatted at 2026-09-27 22:12:29 on dist-test-slave-mp03"
server_key: "de62939866d7bbef30bf4b5a87176dc0"
server_key_iv: "2bcc5a51a7296d0a00acdaa3bde35404"
server_key_version: "encryptionkey@0"
I20260927 22:12:29.649255 15875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "e40f63c65f4f49858110085b873b4028"
format_stamp: "Formatted at 2026-09-27 22:12:29 on dist-test-slave-mp03"
server_key: "de62939866d7bbef30bf4b5a87176dc0"
server_key_iv: "2bcc5a51a7296d0a00acdaa3bde35404"
server_key_version: "encryptionkey@0"
I20260927 22:12:29.655323 15875 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260927 22:12:29.658892 15890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:29.660302 15875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:12:29.660465 15875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "e40f63c65f4f49858110085b873b4028"
format_stamp: "Formatted at 2026-09-27 22:12:29 on dist-test-slave-mp03"
server_key: "de62939866d7bbef30bf4b5a87176dc0"
server_key_iv: "2bcc5a51a7296d0a00acdaa3bde35404"
server_key_version: "encryptionkey@0"
I20260927 22:12:29.660657 15875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:12:29.683558 15875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:12:29.688202 15875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:12:29.688447 15875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:12:29.707160 15875 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32915
I20260927 22:12:29.707183 15941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32915 every 8 connection(s)
I20260927 22:12:29.708570 15875 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:12:29.712488 15942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:12:29.716307 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 15875
I20260927 22:12:29.716534 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:12:29.717290 388 external_mini_cluster.cc:1428] Setting key f448b9b24cfd91c51a956170ad3d47ea
I20260927 22:12:29.726564 15942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028: Bootstrap starting.
I20260927 22:12:29.730626 15942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028: Neither blocks nor log segments found. Creating new log.
I20260927 22:12:29.732050 15942 log.cc:824] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:12:29 dist-test-slave-mp03 krb5kdc[15862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:12:29.735371 15942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028: No bootstrap required, opened a new log
I20260927 22:12:29.741830 15945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:29.719487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33042 (local address 127.0.97.62:32915)
0927 22:12:29.720274 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:29.720287 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:29.720330 (+ 43us) server_negotiation.cc:407] Connection header received
0927 22:12:29.722043 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:29.722070 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:29.722560 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:29.723013 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:29.725032 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:29.726398 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:29.728305 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:29.729000 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:29.729138 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:29.734125 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:29.734134 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:29.734150 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:29.734193 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:29.737515 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:29.738315 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:29.738321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:29.738327 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:29.738503 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:29.738967 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:29.738971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:29.738973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:29.739527 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:29.739868 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:29.740339 (+ 471us) server_negotiation.cc:299] Negotiation successful
0927 22:12:29.740776 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":374,"threads_started":1}
I20260927 22:12:29.743813 15942 raft_consensus.cc:348] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f63c65f4f49858110085b873b4028" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32915 } }
I20260927 22:12:29.744194 15942 raft_consensus.cc:374] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:12:29.744316 15942 raft_consensus.cc:729] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e40f63c65f4f49858110085b873b4028, State: Initialized, Role: FOLLOWER
I20260927 22:12:29.744985 15942 consensus_queue.cc:260] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f63c65f4f49858110085b873b4028" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32915 } }
I20260927 22:12:29.745219 15942 raft_consensus.cc:388] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:12:29.745344 15942 raft_consensus.cc:482] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:12:29.745532 15942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:12:29.747817 15942 raft_consensus.cc:504] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f63c65f4f49858110085b873b4028" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32915 } }
I20260927 22:12:29.748314 15942 leader_election.cc:302] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e40f63c65f4f49858110085b873b4028; no voters:
I20260927 22:12:29.748943 15942 leader_election.cc:288] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:12:29.749104 15947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:12:29.749931 15947 raft_consensus.cc:686] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [term 1 LEADER]: Becoming Leader. State: Replica: e40f63c65f4f49858110085b873b4028, State: Running, Role: LEADER
I20260927 22:12:29.750449 15947 consensus_queue.cc:237] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f63c65f4f49858110085b873b4028" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32915 } }
I20260927 22:12:29.751111 15942 sys_catalog.cc:564] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:12:29.753499 15949 sys_catalog.cc:455] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e40f63c65f4f49858110085b873b4028. Latest consensus state: current_term: 1 leader_uuid: "e40f63c65f4f49858110085b873b4028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f63c65f4f49858110085b873b4028" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32915 } } }
I20260927 22:12:29.753968 15949 sys_catalog.cc:458] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [sys.catalog]: This master's current role is: LEADER
I20260927 22:12:29.754133 15948 sys_catalog.cc:455] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e40f63c65f4f49858110085b873b4028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f63c65f4f49858110085b873b4028" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32915 } } }
I20260927 22:12:29.754488 15948 sys_catalog.cc:458] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [sys.catalog]: This master's current role is: LEADER
I20260927 22:12:29.756379 15952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:12:29.761824 15952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:12:29.771531 15952 catalog_manager.cc:1347] Generated new cluster ID: 3a588df17c814a06b3b3ad7059c9f328
I20260927 22:12:29.771667 15952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:12:29.785586 15952 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:12:29.787175 15952 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:12:29.796065 15952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028: Generated new TSK 0
I20260927 22:12:29.796983 15952 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:12:29.864698 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32915
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:12:30.217682 15968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:12:30.217976 15968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:12:30.218070 15968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:12:30.225454 15968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:12:30.225574 15968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:12:30.225689 15968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:12:30.235710 15968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:12:30.236971 15968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:12:30.238282 15968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:12:30.252416 15973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:30.252450 15976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:30.252511 15974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:12:30.253120 15968 server_base.cc:1034] running on GCE node
I20260927 22:12:30.253822 15968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:12:30.254828 15968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:12:30.256053 15968 hybrid_clock.cc:648] HybridClock initialized: now 1790547150256013 us; error 34 us; skew 500 ppm
I20260927 22:12:30.256414 15968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:12:30 dist-test-slave-mp03 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:12:30.259938 15968 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:12:30.261510 15968 webserver.cc:460] Webserver started at http://127.0.97.1:45725/ using document root <none> and password file <none>
I20260927 22:12:30.262217 15968 fs_manager.cc:359] Metadata directory not provided
I20260927 22:12:30.262395 15968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:12:30.262802 15968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:12:30.265957 15968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "7849ea4e869a454bb71b54a3b127eb41"
format_stamp: "Formatted at 2026-09-27 22:12:30 on dist-test-slave-mp03"
server_key: "ad846b32c8ed7da503121197de244248"
server_key_iv: "b3c2e42310b0a17d97886332ed7ef086"
server_key_version: "encryptionkey@0"
I20260927 22:12:30.266830 15968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7849ea4e869a454bb71b54a3b127eb41"
format_stamp: "Formatted at 2026-09-27 22:12:30 on dist-test-slave-mp03"
server_key: "ad846b32c8ed7da503121197de244248"
server_key_iv: "b3c2e42310b0a17d97886332ed7ef086"
server_key_version: "encryptionkey@0"
I20260927 22:12:30.272305 15968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:12:30.276185 15983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:30.277573 15968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:12:30.277765 15968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "7849ea4e869a454bb71b54a3b127eb41"
format_stamp: "Formatted at 2026-09-27 22:12:30 on dist-test-slave-mp03"
server_key: "ad846b32c8ed7da503121197de244248"
server_key_iv: "b3c2e42310b0a17d97886332ed7ef086"
server_key_version: "encryptionkey@0"
I20260927 22:12:30.277998 15968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:12:30.299276 15968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:12:30.303764 15968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:12:30.304127 15968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:12:30.305214 15968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:12:30.306701 15968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:12:30.306810 15968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:30.306985 15968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:12:30.307065 15968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:30.343195 15968 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35883
I20260927 22:12:30.343281 16095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35883 every 8 connection(s)
I20260927 22:12:30.344555 15968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:12:30.352839 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 15968
I20260927 22:12:30.353178 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:12:30.354003 388 external_mini_cluster.cc:1428] Setting key 87ae4118e2c7578f29383bbdf40e6862
Sep 27 22:12:30 dist-test-slave-mp03 krb5kdc[15862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:12:30.368726 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:30.347383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60203 (local address 127.0.97.62:32915)
0927 22:12:30.347849 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:30.347856 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:30.349134 (+ 1278us) server_negotiation.cc:407] Connection header received
0927 22:12:30.351126 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:30.351131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:30.351202 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:30.351307 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:30.353207 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:30.354398 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:30.356400 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:30.357076 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:30.357252 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:30.361086 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:30.361096 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:30.361104 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:30.361156 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:30.364717 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:30.365935 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:30.365940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:30.365942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:30.366089 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:30.366730 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:30.366739 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:30.366742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:30.367076 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:30.367202 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:30.368238 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0927 22:12:30.368464 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":221,"threads_started":1}
I20260927 22:12:30.370334 16099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:30.348059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.1:60203)
0927 22:12:30.348822 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:30.349001 (+ 179us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:30.350847 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:30.351663 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:30.351672 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:30.352276 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:12:30.352987 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:30.353006 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:30.354555 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:30.354559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:30.356253 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:30.356266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:30.357385 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:30.357391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:30.357661 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:30.358411 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:12:30.358439 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:12:30.360786 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:12:30.364985 (+ 4199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:30.364999 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:30.365022 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:30.365755 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:30.366209 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:30.366218 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:30.366225 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:30.366513 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:30.367220 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:12:30.367232 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:12:30.367759 (+ 527us) client_negotiation.cc:769] Sending connection context
0927 22:12:30.368110 (+ 351us) client_negotiation.cc:240] Negotiation successful
0927 22:12:30.368762 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":305,"threads_started":1}
I20260927 22:12:30.372165 16096 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32915
I20260927 22:12:30.372521 16096 heartbeater.cc:464] Registering TS with master...
I20260927 22:12:30.373308 16096 heartbeater.cc:511] Master 127.0.97.62:32915 requested a full tablet report, sending...
I20260927 22:12:30.375826 15907 ts_manager.cc:194] Registered new tserver with Master: 7849ea4e869a454bb71b54a3b127eb41 (127.0.97.1:35883)
I20260927 22:12:30.377347 15907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60203
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:12:30.410853 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32915
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:12:30.758760 16102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:12:30.759128 16102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:12:30.759310 16102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:12:30.767944 16102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:12:30.768123 16102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:12:30.768332 16102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:12:30.779196 16102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:12:30.780521 16102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:12:30.781920 16102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:12:30.795433 16108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:30.795583 16107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:30.795616 16110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:12:30.796541 16102 server_base.cc:1034] running on GCE node
I20260927 22:12:30.797230 16102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:12:30.798197 16102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:12:30.799427 16102 hybrid_clock.cc:648] HybridClock initialized: now 1790547150799383 us; error 33 us; skew 500 ppm
I20260927 22:12:30.799729 16102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:12:30 dist-test-slave-mp03 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:12:30.803174 16102 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:12:30.804770 16102 webserver.cc:460] Webserver started at http://127.0.97.2:45859/ using document root <none> and password file <none>
I20260927 22:12:30.805469 16102 fs_manager.cc:359] Metadata directory not provided
I20260927 22:12:30.805572 16102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:12:30.805837 16102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:12:30.808815 16102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "6a7bf25b4fc5470184bc2f1f7c530311"
format_stamp: "Formatted at 2026-09-27 22:12:30 on dist-test-slave-mp03"
server_key: "380b904cf602a8d612b1a3403aaa1ea1"
server_key_iv: "8eb7b6471c652a9281a92536eaa61a7f"
server_key_version: "encryptionkey@0"
I20260927 22:12:30.809522 16102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6a7bf25b4fc5470184bc2f1f7c530311"
format_stamp: "Formatted at 2026-09-27 22:12:30 on dist-test-slave-mp03"
server_key: "380b904cf602a8d612b1a3403aaa1ea1"
server_key_iv: "8eb7b6471c652a9281a92536eaa61a7f"
server_key_version: "encryptionkey@0"
I20260927 22:12:30.815006 16102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260927 22:12:30.818773 16117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:30.820240 16102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:12:30.820420 16102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "6a7bf25b4fc5470184bc2f1f7c530311"
format_stamp: "Formatted at 2026-09-27 22:12:30 on dist-test-slave-mp03"
server_key: "380b904cf602a8d612b1a3403aaa1ea1"
server_key_iv: "8eb7b6471c652a9281a92536eaa61a7f"
server_key_version: "encryptionkey@0"
I20260927 22:12:30.820619 16102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:12:30.846735 16102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:12:30.851579 16102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:12:30.851888 16102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:12:30.852909 16102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:12:30.854362 16102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:12:30.854456 16102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:30.854619 16102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:12:30.854671 16102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 22:12:30.889232 16102 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33355
I20260927 22:12:30.889282 16229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33355 every 8 connection(s)
I20260927 22:12:30.890573 16102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:12:30.895488 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 16102
I20260927 22:12:30.895699 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:12:30.896241 388 external_mini_cluster.cc:1428] Setting key 1221ba66dc2882fc389b896a1080348b
Sep 27 22:12:30 dist-test-slave-mp03 krb5kdc[15862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:12:30.914824 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:30.893519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34247 (local address 127.0.97.62:32915)
0927 22:12:30.894003 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:30.894010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:30.895160 (+ 1150us) server_negotiation.cc:407] Connection header received
0927 22:12:30.897255 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:30.897259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:30.897384 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:30.897510 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:30.899248 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:30.900501 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:30.902230 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:30.903053 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:30.903208 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:30.907107 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:30.907118 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:30.907126 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:30.907178 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:30.911036 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:30.911996 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:30.912005 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:30.912011 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:30.912188 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:30.912657 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:30.912666 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:30.912672 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:30.913064 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:30.913220 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:30.914013 (+ 793us) server_negotiation.cc:299] Negotiation successful
0927 22:12:30.914278 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":226,"threads_started":1}
I20260927 22:12:30.915791 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:30.894181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.2:34247)
0927 22:12:30.894995 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:30.895035 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:30.896920 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:30.897715 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:30.897725 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:30.898394 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:12:30.899066 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:30.899083 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:30.900654 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:30.900657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:30.902080 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:30.902092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:30.903279 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:30.903284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:30.903497 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:30.904254 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:12:30.904279 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:12:30.906798 (+ 2519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:12:30.911257 (+ 4459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:30.911265 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:30.911281 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:30.911785 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:30.912309 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:30.912313 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:30.912317 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:30.912535 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:30.913228 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:12:30.913236 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:12:30.913612 (+ 376us) client_negotiation.cc:769] Sending connection context
0927 22:12:30.913863 (+ 251us) client_negotiation.cc:240] Negotiation successful
0927 22:12:30.914491 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":354,"threads_started":1}
I20260927 22:12:30.917497 16230 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32915
I20260927 22:12:30.917869 16230 heartbeater.cc:464] Registering TS with master...
I20260927 22:12:30.918684 16230 heartbeater.cc:511] Master 127.0.97.62:32915 requested a full tablet report, sending...
I20260927 22:12:30.920241 15907 ts_manager.cc:194] Registered new tserver with Master: 6a7bf25b4fc5470184bc2f1f7c530311 (127.0.97.2:33355)
I20260927 22:12:30.921096 15907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34247
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:12:30.953990 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32915
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:12:31.285045 16236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:12:31.285434 16236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:12:31.285616 16236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:12:31.293092 16236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:12:31.293215 16236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:12:31.293408 16236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:12:31.303757 16236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:12:31.305136 16236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:12:31.306546 16236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:12:31.320262 16242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:31.320364 16241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:31.320330 16244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:12:31.321578 16236 server_base.cc:1034] running on GCE node
I20260927 22:12:31.322414 16236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:12:31.323650 16236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:12:31.324926 16236 hybrid_clock.cc:648] HybridClock initialized: now 1790547151324870 us; error 71 us; skew 500 ppm
I20260927 22:12:31.325385 16236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:12:31 dist-test-slave-mp03 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:12:31.336596 16236 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:12:31.338331 16236 webserver.cc:460] Webserver started at http://127.0.97.3:33819/ using document root <none> and password file <none>
I20260927 22:12:31.339242 16236 fs_manager.cc:359] Metadata directory not provided
I20260927 22:12:31.339432 16236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:12:31.339833 16236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:12:31.343384 16236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "b682cc8bf46e4d209666aced872365b0"
format_stamp: "Formatted at 2026-09-27 22:12:31 on dist-test-slave-mp03"
server_key: "4474b14bb30c9e152a8c674b090159d4"
server_key_iv: "9e3ec151881c1d199a6bd93c73f082f0"
server_key_version: "encryptionkey@0"
I20260927 22:12:31.344254 16236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b682cc8bf46e4d209666aced872365b0"
format_stamp: "Formatted at 2026-09-27 22:12:31 on dist-test-slave-mp03"
server_key: "4474b14bb30c9e152a8c674b090159d4"
server_key_iv: "9e3ec151881c1d199a6bd93c73f082f0"
server_key_version: "encryptionkey@0"
I20260927 22:12:31.350142 16236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 22:12:31.354171 16251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:31.355623 16236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:12:31.355831 16236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "b682cc8bf46e4d209666aced872365b0"
format_stamp: "Formatted at 2026-09-27 22:12:31 on dist-test-slave-mp03"
server_key: "4474b14bb30c9e152a8c674b090159d4"
server_key_iv: "9e3ec151881c1d199a6bd93c73f082f0"
server_key_version: "encryptionkey@0"
I20260927 22:12:31.356070 16236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:12:31.373979 16236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:12:31.379276 16236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:12:31.379649 16236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:12:31.380920 16096 heartbeater.cc:503] Master 127.0.97.62:32915 was elected leader, sending a full tablet report...
I20260927 22:12:31.380998 16236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:12:31.383139 16236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:12:31.383255 16236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:31.383411 16236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:12:31.383484 16236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:31.421309 16236 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45843
I20260927 22:12:31.421320 16363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45843 every 8 connection(s)
I20260927 22:12:31.422895 16236 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:12:31.428097 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 16236
I20260927 22:12:31.428287 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:12:31.428742 388 external_mini_cluster.cc:1428] Setting key 6e5e9b619926b43f00a64d61232b73fe
Sep 27 22:12:31 dist-test-slave-mp03 krb5kdc[15862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:12:31.443181 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:31.425722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48821 (local address 127.0.97.62:32915)
0927 22:12:31.426172 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:31.426178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:31.427286 (+ 1108us) server_negotiation.cc:407] Connection header received
0927 22:12:31.429208 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:31.429215 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:31.429294 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:31.429392 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:31.431091 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.432035 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.433533 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.433931 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.434066 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:31.437440 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:31.437445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:31.437448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:31.437482 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:31.440018 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:31.440726 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:31.440735 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:31.440737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:31.440856 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:31.441291 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:31.441296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:31.441299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:31.441581 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:31.441773 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:31.442505 (+ 732us) server_negotiation.cc:299] Negotiation successful
0927 22:12:31.442771 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":203,"threads_started":1}
I20260927 22:12:31.444487 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:31.426409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.3:48821)
0927 22:12:31.427131 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:31.427167 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:31.428947 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:31.429642 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:31.429649 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:31.430252 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:12:31.430926 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:31.430943 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.432189 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.432194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:31.433390 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:31.433401 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.434148 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.434155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:31.434392 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:31.435100 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:12:31.435122 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:12:31.437229 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:12:31.440160 (+ 2931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:31.440169 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:31.440187 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:31.440598 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:31.440963 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:31.440968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:31.440971 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:31.441174 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:31.441762 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:12:31.441770 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:12:31.442124 (+ 354us) client_negotiation.cc:769] Sending connection context
0927 22:12:31.442410 (+ 286us) client_negotiation.cc:240] Negotiation successful
0927 22:12:31.443010 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":324,"threads_started":1}
I20260927 22:12:31.446030 16364 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32915
I20260927 22:12:31.446305 16364 heartbeater.cc:464] Registering TS with master...
I20260927 22:12:31.447053 16364 heartbeater.cc:511] Master 127.0.97.62:32915 requested a full tablet report, sending...
I20260927 22:12:31.448453 15907 ts_manager.cc:194] Registered new tserver with Master: b682cc8bf46e4d209666aced872365b0 (127.0.97.3:45843)
I20260927 22:12:31.449275 15907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48821
I20260927 22:12:31.457147 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:12:31.474570 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:31.461633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33044 (local address 127.0.97.62:32915)
0927 22:12:31.461818 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:31.461826 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:31.462227 (+ 401us) server_negotiation.cc:407] Connection header received
0927 22:12:31.462418 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:31.462425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:31.462500 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:31.462604 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:31.464213 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.465050 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.466399 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.466859 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.467004 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:31.469346 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:31.469351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:31.469355 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:31.469388 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:31.471922 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:31.472578 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:31.472585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:31.472593 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:31.472704 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:31.473084 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:31.473090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:31.473093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:31.473420 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:31.473688 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:31.473991 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 22:12:31.474223 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:12:31.478675 15907 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:12:31.486003 15907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:12:31.510902 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:31.499171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45843 (local address 127.0.0.1:42170)
0927 22:12:31.500316 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:31.500366 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:31.500589 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:31.503496 (+ 2907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:31.503504 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:31.503543 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:12:31.504011 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:31.504022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.506120 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.506125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:31.507946 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:31.507965 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.509767 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.509773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:31.509982 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:31.510055 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 22:12:31.510177 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 22:12:31.510370 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":30,"thread_start_us":1736,"threads_started":1}
I20260927 22:12:31.511389 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:31.499513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58672 (local address 127.0.97.1:35883)
0927 22:12:31.502250 (+ 2737us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:31.502257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:31.502281 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:12:31.502388 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:31.502399 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:31.502707 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:31.502921 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:12:31.503667 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.505306 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.508189 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.509568 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.509687 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:31.509930 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:31.510894 (+ 964us) server_negotiation.cc:299] Negotiation successful
0927 22:12:31.511081 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2582,"thread_start_us":2418,"threads_started":1}
I20260927 22:12:31.511442 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:31.499690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42170 (local address 127.0.97.3:45843)
0927 22:12:31.502863 (+ 3173us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:31.502874 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:31.502911 (+ 37us) server_negotiation.cc:407] Connection header received
0927 22:12:31.503023 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:31.503033 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:31.503266 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:31.504211 (+ 945us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:12:31.504425 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.505977 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.508104 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.509542 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.510559 (+ 1017us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:31.510809 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:31.510880 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 22:12:31.511057 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2964,"thread_start_us":1823,"threads_started":1}
I20260927 22:12:31.511747 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:31.499214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35883 (local address 127.0.0.1:58672)
0927 22:12:31.501170 (+ 1956us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:31.501185 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:31.501281 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:31.502922 (+ 1641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:31.502929 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:31.502963 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:12:31.503459 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:31.503479 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.505473 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.505478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:31.507989 (+ 2511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:31.508006 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.510082 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.510088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:31.510267 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:31.510296 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 22:12:31.510368 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 22:12:31.510489 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1825,"mutex_wait_us":59,"thread_start_us":786,"threads_started":1}
I20260927 22:12:31.511958 16381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:31.501791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33355 (local address 127.0.0.1:40072)
0927 22:12:31.503626 (+ 1835us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:31.503642 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:31.503752 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:31.504794 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:31.504800 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:31.504826 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:12:31.505174 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:31.505185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.507016 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.507021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:31.508775 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:31.508785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.510628 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.510635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:31.510813 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:31.510861 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 22:12:31.511383 (+ 522us) client_negotiation.cc:240] Negotiation successful
0927 22:12:31.511520 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1693,"mutex_wait_us":1,"thread_start_us":1524,"threads_started":1}
I20260927 22:12:31.512607 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:31.502273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40072 (local address 127.0.97.2:33355)
0927 22:12:31.504212 (+ 1939us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:31.504222 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:31.504248 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:12:31.504344 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:31.504350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:31.504546 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:31.504805 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:12:31.505353 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.506865 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.508924 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.510105 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.510232 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:31.510436 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:31.512137 (+ 1701us) server_negotiation.cc:299] Negotiation successful
0927 22:12:31.512316 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1793,"thread_start_us":1686,"threads_started":1}
I20260927 22:12:31.515587 16299 tablet_service.cc:1467] Processing CreateTablet for tablet 277f4054e2cf40fc8cdaf78e99b6635a (DEFAULT_TABLE table=test-table [id=1926c1df2ecf49d28812d2fe272f2e4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:12:31.515687 16031 tablet_service.cc:1467] Processing CreateTablet for tablet 277f4054e2cf40fc8cdaf78e99b6635a (DEFAULT_TABLE table=test-table [id=1926c1df2ecf49d28812d2fe272f2e4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:12:31.515843 16165 tablet_service.cc:1467] Processing CreateTablet for tablet 277f4054e2cf40fc8cdaf78e99b6635a (DEFAULT_TABLE table=test-table [id=1926c1df2ecf49d28812d2fe272f2e4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:12:31.516920 16299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 277f4054e2cf40fc8cdaf78e99b6635a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:12:31.517030 16031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 277f4054e2cf40fc8cdaf78e99b6635a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:12:31.517030 16165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 277f4054e2cf40fc8cdaf78e99b6635a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:12:31.530230 16383 tablet_bootstrap.cc:492] T 277f4054e2cf40fc8cdaf78e99b6635a P b682cc8bf46e4d209666aced872365b0: Bootstrap starting.
I20260927 22:12:31.532195 16384 tablet_bootstrap.cc:492] T 277f4054e2cf40fc8cdaf78e99b6635a P 6a7bf25b4fc5470184bc2f1f7c530311: Bootstrap starting.
I20260927 22:12:31.534169 16383 tablet_bootstrap.cc:654] T 277f4054e2cf40fc8cdaf78e99b6635a P b682cc8bf46e4d209666aced872365b0: Neither blocks nor log segments found. Creating new log.
I20260927 22:12:31.535513 16383 log.cc:824] T 277f4054e2cf40fc8cdaf78e99b6635a P b682cc8bf46e4d209666aced872365b0: Log is configured to *not* fsync() on all Append() calls
I20260927 22:12:31.535761 16384 tablet_bootstrap.cc:654] T 277f4054e2cf40fc8cdaf78e99b6635a P 6a7bf25b4fc5470184bc2f1f7c530311: Neither blocks nor log segments found. Creating new log.
I20260927 22:12:31.537009 16384 log.cc:824] T 277f4054e2cf40fc8cdaf78e99b6635a P 6a7bf25b4fc5470184bc2f1f7c530311: Log is configured to *not* fsync() on all Append() calls
I20260927 22:12:31.536979 16385 tablet_bootstrap.cc:492] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41: Bootstrap starting.
I20260927 22:12:31.538976 16383 tablet_bootstrap.cc:492] T 277f4054e2cf40fc8cdaf78e99b6635a P b682cc8bf46e4d209666aced872365b0: No bootstrap required, opened a new log
I20260927 22:12:31.539273 16383 ts_tablet_manager.cc:1397] T 277f4054e2cf40fc8cdaf78e99b6635a P b682cc8bf46e4d209666aced872365b0: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260927 22:12:31.540299 16384 tablet_bootstrap.cc:492] T 277f4054e2cf40fc8cdaf78e99b6635a P 6a7bf25b4fc5470184bc2f1f7c530311: No bootstrap required, opened a new log
I20260927 22:12:31.540576 16384 ts_tablet_manager.cc:1397] T 277f4054e2cf40fc8cdaf78e99b6635a P 6a7bf25b4fc5470184bc2f1f7c530311: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 22:12:31.541550 16385 tablet_bootstrap.cc:654] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41: Neither blocks nor log segments found. Creating new log.
I20260927 22:12:31.542829 16385 log.cc:824] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41: Log is configured to *not* fsync() on all Append() calls
I20260927 22:12:31.545895 16385 tablet_bootstrap.cc:492] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41: No bootstrap required, opened a new log
I20260927 22:12:31.546168 16385 ts_tablet_manager.cc:1397] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260927 22:12:31.547014 16383 raft_consensus.cc:348] T 277f4054e2cf40fc8cdaf78e99b6635a P b682cc8bf46e4d209666aced872365b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b682cc8bf46e4d209666aced872365b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45843 } } peers { permanent_uuid: "7849ea4e869a454bb71b54a3b127eb41" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35883 } } peers { permanent_uuid: "6a7bf25b4fc5470184bc2f1f7c530311" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } }
I20260927 22:12:31.547384 16383 raft_consensus.cc:374] T 277f4054e2cf40fc8cdaf78e99b6635a P b682cc8bf46e4d209666aced872365b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:12:31.547516 16383 raft_consensus.cc:729] T 277f4054e2cf40fc8cdaf78e99b6635a P b682cc8bf46e4d209666aced872365b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b682cc8bf46e4d209666aced872365b0, State: Initialized, Role: FOLLOWER
I20260927 22:12:31.548303 16383 consensus_queue.cc:260] T 277f4054e2cf40fc8cdaf78e99b6635a P b682cc8bf46e4d209666aced872365b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b682cc8bf46e4d209666aced872365b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45843 } } peers { permanent_uuid: "7849ea4e869a454bb71b54a3b127eb41" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35883 } } peers { permanent_uuid: "6a7bf25b4fc5470184bc2f1f7c530311" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } }
I20260927 22:12:31.550850 16384 raft_consensus.cc:348] T 277f4054e2cf40fc8cdaf78e99b6635a P 6a7bf25b4fc5470184bc2f1f7c530311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b682cc8bf46e4d209666aced872365b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45843 } } peers { permanent_uuid: "7849ea4e869a454bb71b54a3b127eb41" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35883 } } peers { permanent_uuid: "6a7bf25b4fc5470184bc2f1f7c530311" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } }
I20260927 22:12:31.551375 16384 raft_consensus.cc:374] T 277f4054e2cf40fc8cdaf78e99b6635a P 6a7bf25b4fc5470184bc2f1f7c530311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:12:31.551388 16364 heartbeater.cc:503] Master 127.0.97.62:32915 was elected leader, sending a full tablet report...
I20260927 22:12:31.551571 16384 raft_consensus.cc:729] T 277f4054e2cf40fc8cdaf78e99b6635a P 6a7bf25b4fc5470184bc2f1f7c530311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a7bf25b4fc5470184bc2f1f7c530311, State: Initialized, Role: FOLLOWER
I20260927 22:12:31.552132 16383 ts_tablet_manager.cc:1428] T 277f4054e2cf40fc8cdaf78e99b6635a P b682cc8bf46e4d209666aced872365b0: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260927 22:12:31.552418 16384 consensus_queue.cc:260] T 277f4054e2cf40fc8cdaf78e99b6635a P 6a7bf25b4fc5470184bc2f1f7c530311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b682cc8bf46e4d209666aced872365b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45843 } } peers { permanent_uuid: "7849ea4e869a454bb71b54a3b127eb41" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35883 } } peers { permanent_uuid: "6a7bf25b4fc5470184bc2f1f7c530311" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } }
I20260927 22:12:31.553665 16230 heartbeater.cc:503] Master 127.0.97.62:32915 was elected leader, sending a full tablet report...
I20260927 22:12:31.554353 16384 ts_tablet_manager.cc:1428] T 277f4054e2cf40fc8cdaf78e99b6635a P 6a7bf25b4fc5470184bc2f1f7c530311: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260927 22:12:31.556717 16385 raft_consensus.cc:348] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b682cc8bf46e4d209666aced872365b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45843 } } peers { permanent_uuid: "7849ea4e869a454bb71b54a3b127eb41" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35883 } } peers { permanent_uuid: "6a7bf25b4fc5470184bc2f1f7c530311" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } }
I20260927 22:12:31.557186 16385 raft_consensus.cc:374] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:12:31.557296 16385 raft_consensus.cc:729] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7849ea4e869a454bb71b54a3b127eb41, State: Initialized, Role: FOLLOWER
I20260927 22:12:31.557991 16385 consensus_queue.cc:260] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b682cc8bf46e4d209666aced872365b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45843 } } peers { permanent_uuid: "7849ea4e869a454bb71b54a3b127eb41" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35883 } } peers { permanent_uuid: "6a7bf25b4fc5470184bc2f1f7c530311" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } }
I20260927 22:12:31.559729 16385 ts_tablet_manager.cc:1428] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41: Time spent starting tablet: real 0.013s user 0.007s sys 0.007s
W20260927 22:12:31.597339 16097 tablet.cc:2369] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:12:31.642747 16231 tablet.cc:2369] T 277f4054e2cf40fc8cdaf78e99b6635a P 6a7bf25b4fc5470184bc2f1f7c530311: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:12:31.674825 16365 tablet.cc:2369] T 277f4054e2cf40fc8cdaf78e99b6635a P b682cc8bf46e4d209666aced872365b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:12:31.680606 16391 raft_consensus.cc:482] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:12:31.680981 16391 raft_consensus.cc:504] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b682cc8bf46e4d209666aced872365b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45843 } } peers { permanent_uuid: "7849ea4e869a454bb71b54a3b127eb41" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35883 } } peers { permanent_uuid: "6a7bf25b4fc5470184bc2f1f7c530311" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } }
I20260927 22:12:31.682857 16391 leader_election.cc:288] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b682cc8bf46e4d209666aced872365b0 (127.0.97.3:45843), 6a7bf25b4fc5470184bc2f1f7c530311 (127.0.97.2:33355)
I20260927 22:12:31.689751 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:31.683353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35445 (local address 127.0.97.2:33355)
0927 22:12:31.683620 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:31.683629 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:31.684201 (+ 572us) server_negotiation.cc:407] Connection header received
0927 22:12:31.684447 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:31.684454 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:31.684544 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:31.684624 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:12:31.685303 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.686047 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.687981 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.688744 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.688893 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:31.688994 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:31.689350 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 22:12:31.689480 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 22:12:31.689899 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:31.683358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54323 (local address 127.0.97.3:45843)
0927 22:12:31.683812 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:31.683817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:31.684353 (+ 536us) server_negotiation.cc:407] Connection header received
0927 22:12:31.684528 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:31.684532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:31.684631 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:31.684721 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:12:31.685377 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.686141 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.688018 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.688819 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.688935 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:31.689051 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:31.689525 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 22:12:31.689657 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304}
I20260927 22:12:31.690229 16392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:31.683107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33355 (local address 127.0.97.1:35445)
0927 22:12:31.684034 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:31.684056 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:31.684260 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:31.684677 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:31.684684 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:31.684713 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:12:31.685163 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:31.685173 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.686184 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.686192 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:31.687829 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:31.687840 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.688973 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.688979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:31.689159 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:31.689206 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 22:12:31.689326 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 22:12:31.689456 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":2,"spinlock_wait_cycles":3840,"thread_start_us":531,"threads_started":1}
I20260927 22:12:31.690686 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:31.683357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45843 (local address 127.0.97.1:54323)
0927 22:12:31.684230 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:31.684247 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:31.684393 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:31.684761 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:31.684771 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:31.684818 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:12:31.685192 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:31.685200 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.686273 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.686276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:31.687836 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:31.687847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.689153 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.689158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:31.689325 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:31.689357 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:12:31.690020 (+ 663us) client_negotiation.cc:240] Negotiation successful
0927 22:12:31.690171 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":58,"spinlock_wait_cycles":16640,"thread_start_us":450,"threads_started":1}
I20260927 22:12:31.691146 16185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "277f4054e2cf40fc8cdaf78e99b6635a" candidate_uuid: "7849ea4e869a454bb71b54a3b127eb41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a7bf25b4fc5470184bc2f1f7c530311" is_pre_election: true
I20260927 22:12:31.691785 16185 raft_consensus.cc:2454] T 277f4054e2cf40fc8cdaf78e99b6635a P 6a7bf25b4fc5470184bc2f1f7c530311 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7849ea4e869a454bb71b54a3b127eb41 in term 0.
I20260927 22:12:31.691751 16319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "277f4054e2cf40fc8cdaf78e99b6635a" candidate_uuid: "7849ea4e869a454bb71b54a3b127eb41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b682cc8bf46e4d209666aced872365b0" is_pre_election: true
I20260927 22:12:31.692266 16319 raft_consensus.cc:2454] T 277f4054e2cf40fc8cdaf78e99b6635a P b682cc8bf46e4d209666aced872365b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7849ea4e869a454bb71b54a3b127eb41 in term 0.
I20260927 22:12:31.692520 15986 leader_election.cc:302] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a7bf25b4fc5470184bc2f1f7c530311, 7849ea4e869a454bb71b54a3b127eb41; no voters:
I20260927 22:12:31.693005 16391 raft_consensus.cc:2781] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:12:31.693135 16391 raft_consensus.cc:482] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:12:31.693272 16391 raft_consensus.cc:3037] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:12:31.695688 16391 raft_consensus.cc:504] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b682cc8bf46e4d209666aced872365b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45843 } } peers { permanent_uuid: "7849ea4e869a454bb71b54a3b127eb41" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35883 } } peers { permanent_uuid: "6a7bf25b4fc5470184bc2f1f7c530311" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } }
I20260927 22:12:31.696348 16391 leader_election.cc:288] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41 [CANDIDATE]: Term 1 election: Requested vote from peers b682cc8bf46e4d209666aced872365b0 (127.0.97.3:45843), 6a7bf25b4fc5470184bc2f1f7c530311 (127.0.97.2:33355)
I20260927 22:12:31.696796 16319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "277f4054e2cf40fc8cdaf78e99b6635a" candidate_uuid: "7849ea4e869a454bb71b54a3b127eb41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b682cc8bf46e4d209666aced872365b0"
I20260927 22:12:31.696913 16185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "277f4054e2cf40fc8cdaf78e99b6635a" candidate_uuid: "7849ea4e869a454bb71b54a3b127eb41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a7bf25b4fc5470184bc2f1f7c530311"
I20260927 22:12:31.697062 16319 raft_consensus.cc:3037] T 277f4054e2cf40fc8cdaf78e99b6635a P b682cc8bf46e4d209666aced872365b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:12:31.697119 16185 raft_consensus.cc:3037] T 277f4054e2cf40fc8cdaf78e99b6635a P 6a7bf25b4fc5470184bc2f1f7c530311 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:12:31.700250 16319 raft_consensus.cc:2454] T 277f4054e2cf40fc8cdaf78e99b6635a P b682cc8bf46e4d209666aced872365b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7849ea4e869a454bb71b54a3b127eb41 in term 1.
I20260927 22:12:31.700250 16185 raft_consensus.cc:2454] T 277f4054e2cf40fc8cdaf78e99b6635a P 6a7bf25b4fc5470184bc2f1f7c530311 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7849ea4e869a454bb71b54a3b127eb41 in term 1.
I20260927 22:12:31.700696 15986 leader_election.cc:302] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6a7bf25b4fc5470184bc2f1f7c530311, 7849ea4e869a454bb71b54a3b127eb41, b682cc8bf46e4d209666aced872365b0; no voters:
I20260927 22:12:31.701188 16391 raft_consensus.cc:2781] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:12:31.701679 16391 raft_consensus.cc:686] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41 [term 1 LEADER]: Becoming Leader. State: Replica: 7849ea4e869a454bb71b54a3b127eb41, State: Running, Role: LEADER
I20260927 22:12:31.702162 16391 consensus_queue.cc:237] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b682cc8bf46e4d209666aced872365b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45843 } } peers { permanent_uuid: "7849ea4e869a454bb71b54a3b127eb41" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35883 } } peers { permanent_uuid: "6a7bf25b4fc5470184bc2f1f7c530311" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } }
I20260927 22:12:31.707242 15907 catalog_manager.cc:5495] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7849ea4e869a454bb71b54a3b127eb41 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "7849ea4e869a454bb71b54a3b127eb41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b682cc8bf46e4d209666aced872365b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7849ea4e869a454bb71b54a3b127eb41" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a7bf25b4fc5470184bc2f1f7c530311" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } health_report { overall_health: UNKNOWN } } }
I20260927 22:12:31.747921 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:31.742882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58680 (local address 127.0.97.1:35883)
0927 22:12:31.743103 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:31.743108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:31.743124 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:12:31.743214 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:31.743218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:31.743295 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:31.743423 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:12:31.743786 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.744518 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.745874 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.746275 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.746364 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:31.746876 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:12:31.747450 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:12:31.747569 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:31.747657 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 22:12:31.747722 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 22:12:31.756871 16319 raft_consensus.cc:1261] T 277f4054e2cf40fc8cdaf78e99b6635a P b682cc8bf46e4d209666aced872365b0 [term 1 FOLLOWER]: Refusing update from remote peer 7849ea4e869a454bb71b54a3b127eb41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:12:31.756871 16185 raft_consensus.cc:1261] T 277f4054e2cf40fc8cdaf78e99b6635a P 6a7bf25b4fc5470184bc2f1f7c530311 [term 1 FOLLOWER]: Refusing update from remote peer 7849ea4e869a454bb71b54a3b127eb41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:12:31.757941 16394 consensus_queue.cc:1035] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b682cc8bf46e4d209666aced872365b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:12:31.758284 16391 consensus_queue.cc:1035] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a7bf25b4fc5470184bc2f1f7c530311" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:12:31.770408 16397 mvcc.cc:205] Tried to move back new op lower bound from 7334081133584109568 to 7334081133376483328. Current Snapshot: MvccSnapshot[applied={T|T < 7334081133584109568}]
I20260927 22:12:31.784493 15906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:12:31.784861 15906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:12:31.788743 15906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028: Sending DeleteTablet for 3 replicas of tablet 277f4054e2cf40fc8cdaf78e99b6635a
I20260927 22:12:31.789999 16299 tablet_service.cc:1514] Processing DeleteTablet for tablet 277f4054e2cf40fc8cdaf78e99b6635a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:12:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42170
I20260927 22:12:31.790155 16165 tablet_service.cc:1514] Processing DeleteTablet for tablet 277f4054e2cf40fc8cdaf78e99b6635a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:12:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40072
I20260927 22:12:31.790289 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 15968
I20260927 22:12:31.790352 16031 tablet_service.cc:1514] Processing DeleteTablet for tablet 277f4054e2cf40fc8cdaf78e99b6635a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:12:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58672
I20260927 22:12:31.800665 16409 tablet_replica.cc:330] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41: stopping tablet replica
I20260927 22:12:31.801311 16409 raft_consensus.cc:2229] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:12:31.803092 16409 raft_consensus.cc:2258] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:12:31.805629 16409 ts_tablet_manager.cc:1905] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:12:31.806469 16407 tablet_replica.cc:330] T 277f4054e2cf40fc8cdaf78e99b6635a P b682cc8bf46e4d209666aced872365b0: stopping tablet replica
I20260927 22:12:31.806947 16407 raft_consensus.cc:2229] T 277f4054e2cf40fc8cdaf78e99b6635a P b682cc8bf46e4d209666aced872365b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:12:31.807068 16408 tablet_replica.cc:330] T 277f4054e2cf40fc8cdaf78e99b6635a P 6a7bf25b4fc5470184bc2f1f7c530311: stopping tablet replica
I20260927 22:12:31.807488 16407 raft_consensus.cc:2258] T 277f4054e2cf40fc8cdaf78e99b6635a P b682cc8bf46e4d209666aced872365b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:12:31.807651 16408 raft_consensus.cc:2229] T 277f4054e2cf40fc8cdaf78e99b6635a P 6a7bf25b4fc5470184bc2f1f7c530311 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:12:31.808208 16408 raft_consensus.cc:2258] T 277f4054e2cf40fc8cdaf78e99b6635a P 6a7bf25b4fc5470184bc2f1f7c530311 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:12:31.809847 16407 ts_tablet_manager.cc:1905] T 277f4054e2cf40fc8cdaf78e99b6635a P b682cc8bf46e4d209666aced872365b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:12:31.812155 16408 ts_tablet_manager.cc:1905] T 277f4054e2cf40fc8cdaf78e99b6635a P 6a7bf25b4fc5470184bc2f1f7c530311: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 22:12:31 dist-test-slave-mp03 krb5kdc[15862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:12:31.816905 16407 ts_tablet_manager.cc:1918] T 277f4054e2cf40fc8cdaf78e99b6635a P b682cc8bf46e4d209666aced872365b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:12:31.817121 16407 log.cc:1196] T 277f4054e2cf40fc8cdaf78e99b6635a P b682cc8bf46e4d209666aced872365b0: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/277f4054e2cf40fc8cdaf78e99b6635a
I20260927 22:12:31.817646 16407 ts_tablet_manager.cc:1939] T 277f4054e2cf40fc8cdaf78e99b6635a P b682cc8bf46e4d209666aced872365b0: Deleting consensus metadata
I20260927 22:12:31.819665 15892 catalog_manager.cc:4841] TS b682cc8bf46e4d209666aced872365b0 (127.0.97.3:45843): tablet 277f4054e2cf40fc8cdaf78e99b6635a (table test-table [id=1926c1df2ecf49d28812d2fe272f2e4c]) successfully deleted
I20260927 22:12:31.823625 16409 ts_tablet_manager.cc:1918] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:12:31.823886 16409 log.cc:1196] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/wal/wals/277f4054e2cf40fc8cdaf78e99b6635a
I20260927 22:12:31.824486 16409 ts_tablet_manager.cc:1939] T 277f4054e2cf40fc8cdaf78e99b6635a P 7849ea4e869a454bb71b54a3b127eb41: Deleting consensus metadata
I20260927 22:12:31.824926 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:31.791779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58684 (local address 127.0.97.1:35883)
0927 22:12:31.792011 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:31.792021 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:31.802228 (+ 10207us) server_negotiation.cc:407] Connection header received
0927 22:12:31.802439 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:31.802447 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:31.802588 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:31.802744 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:31.806172 (+ 3428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.807339 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.809503 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.810235 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.810361 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:31.816530 (+ 6169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:31.816551 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:31.816571 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:31.816629 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:31.820903 (+ 4274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:31.821913 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:31.821922 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:31.821929 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:31.822120 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:31.822666 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:31.822672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:31.822675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:31.823165 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:31.823476 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:31.824185 (+ 709us) server_negotiation.cc:299] Negotiation successful
0927 22:12:31.824512 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:12:31.824915 16408 ts_tablet_manager.cc:1918] T 277f4054e2cf40fc8cdaf78e99b6635a P 6a7bf25b4fc5470184bc2f1f7c530311: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:12:31.825163 16408 log.cc:1196] T 277f4054e2cf40fc8cdaf78e99b6635a P 6a7bf25b4fc5470184bc2f1f7c530311: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/277f4054e2cf40fc8cdaf78e99b6635a
I20260927 22:12:31.825802 16408 ts_tablet_manager.cc:1939] T 277f4054e2cf40fc8cdaf78e99b6635a P 6a7bf25b4fc5470184bc2f1f7c530311: Deleting consensus metadata
I20260927 22:12:31.826653 16091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:12:31.827891 15893 catalog_manager.cc:4841] TS 6a7bf25b4fc5470184bc2f1f7c530311 (127.0.97.2:33355): tablet 277f4054e2cf40fc8cdaf78e99b6635a (table test-table [id=1926c1df2ecf49d28812d2fe272f2e4c]) successfully deleted
I20260927 22:12:31.927054 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 15968
I20260927 22:12:31.927093 15893 catalog_manager.cc:4841] TS 7849ea4e869a454bb71b54a3b127eb41 (127.0.97.1:35883): tablet 277f4054e2cf40fc8cdaf78e99b6635a (table test-table [id=1926c1df2ecf49d28812d2fe272f2e4c]) successfully deleted
I20260927 22:12:31.946065 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 16102
Sep 27 22:12:31 dist-test-slave-mp03 krb5kdc[15862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:12:31.962306 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:31.946910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40078 (local address 127.0.97.2:33355)
0927 22:12:31.947152 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:31.947156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:31.947178 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:12:31.947358 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:31.947362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:31.947441 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:31.947529 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:31.948743 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.949521 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.950986 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:31.951387 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:31.951484 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:31.956363 (+ 4879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:31.956378 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:31.956394 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:31.956429 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:31.959393 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:31.960231 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:31.960239 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:31.960241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:31.960362 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:31.960782 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:31.960785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:31.960787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:31.961311 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:31.961530 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:31.961843 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 22:12:31.962035 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 22:12:31.963042 16225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:12:32.055032 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 16102
I20260927 22:12:32.072319 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 16236
Sep 27 22:12:32 dist-test-slave-mp03 krb5kdc[15862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:12:32.087435 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:32.073176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.0.97.3:45843)
0927 22:12:32.073373 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:32.073378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:32.073397 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:12:32.073453 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:32.073457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:32.073519 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:32.073615 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:32.074960 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:32.075684 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:32.077044 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:32.077441 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:32.077538 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:32.081890 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:32.081910 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:32.081928 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:32.081976 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:32.084805 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:32.085459 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:32.085466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:32.085468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:32.085620 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:32.085929 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:32.085932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:32.085934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:32.086295 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:32.086553 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:32.086906 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 22:12:32.087071 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:12:32.088146 16359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:12:32.178419 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 16236
I20260927 22:12:32.197125 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 15875
I20260927 22:12:32.198252 15937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:12:32.311681 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 15875
I20260927 22:12:52.333793 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32915
--webserver_interface=127.0.97.62
--webserver_port=39915
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:12:52.672508 16415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:12:52.672854 16415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:12:52.673013 16415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:12:52.680888 16415 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 22:12:52.681036 16415 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 22:12:52.681144 16415 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 22:12:52.681252 16415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:12:52.681367 16415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:12:52.681502 16415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:12:52.681600 16415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:12:52.681689 16415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:12:52.691833 16415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=39915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:12:52.693189 16415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:12:52.694643 16415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:12:52.718101 16424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:52.718163 16422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:52.718127 16421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:12:52.718974 16415 server_base.cc:1034] running on GCE node
I20260927 22:12:52.719903 16415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:12:52.721433 16415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:12:52.722709 16415 hybrid_clock.cc:648] HybridClock initialized: now 1790547172722641 us; error 43 us; skew 500 ppm
I20260927 22:12:52.723098 16415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:12:52 dist-test-slave-mp03 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:12:52.731877 16415 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:12:52.733552 16415 webserver.cc:460] Webserver started at http://127.0.97.62:39915/ using document root <none> and password file <none>
I20260927 22:12:52.734360 16415 fs_manager.cc:359] Metadata directory not provided
I20260927 22:12:52.734519 16415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:12:52.740581 16415 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:12:52.743604 16431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:52.745064 16415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:12:52.745261 16415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "e40f63c65f4f49858110085b873b4028"
format_stamp: "Formatted at 2026-09-27 22:12:29 on dist-test-slave-mp03"
server_key: "de62939866d7bbef30bf4b5a87176dc0"
server_key_iv: "2bcc5a51a7296d0a00acdaa3bde35404"
server_key_version: "encryptionkey@0"
I20260927 22:12:52.746297 16415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:12:52.778829 16415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:12:52.783500 16415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:12:52.783802 16415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:12:52.803887 16415 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32915
I20260927 22:12:52.803929 16482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32915 every 8 connection(s)
I20260927 22:12:52.805627 16415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:12:52.810158 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 16415
I20260927 22:12:52.811174 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:35883
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:32915
--webserver_port=45725
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:12:52.814281 16483 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:12:52.819828 16483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028: Bootstrap starting.
I20260927 22:12:52.840528 16483 log.cc:824] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028: Log is configured to *not* fsync() on all Append() calls
I20260927 22:12:52.853147 16483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:12:52.853714 16483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028: Bootstrap complete.
I20260927 22:12:52.863310 16483 raft_consensus.cc:348] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f63c65f4f49858110085b873b4028" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32915 } }
I20260927 22:12:52.864164 16483 raft_consensus.cc:729] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e40f63c65f4f49858110085b873b4028, State: Initialized, Role: FOLLOWER
I20260927 22:12:52.864858 16483 consensus_queue.cc:260] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f63c65f4f49858110085b873b4028" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32915 } }
I20260927 22:12:52.865110 16483 raft_consensus.cc:388] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:12:52.865232 16483 raft_consensus.cc:482] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:12:52.865424 16483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:12:52.869418 16483 raft_consensus.cc:504] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f63c65f4f49858110085b873b4028" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32915 } }
I20260927 22:12:52.869949 16483 leader_election.cc:302] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e40f63c65f4f49858110085b873b4028; no voters:
I20260927 22:12:52.870699 16483 leader_election.cc:288] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:12:52.870841 16487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:12:52.873221 16483 sys_catalog.cc:564] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:12:52.878067 16487 raft_consensus.cc:686] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [term 2 LEADER]: Becoming Leader. State: Replica: e40f63c65f4f49858110085b873b4028, State: Running, Role: LEADER
I20260927 22:12:52.878907 16487 consensus_queue.cc:237] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f63c65f4f49858110085b873b4028" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32915 } }
I20260927 22:12:52.882215 16488 sys_catalog.cc:455] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e40f63c65f4f49858110085b873b4028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f63c65f4f49858110085b873b4028" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32915 } } }
I20260927 22:12:52.882721 16488 sys_catalog.cc:458] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [sys.catalog]: This master's current role is: LEADER
I20260927 22:12:52.883330 16489 sys_catalog.cc:455] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e40f63c65f4f49858110085b873b4028. Latest consensus state: current_term: 2 leader_uuid: "e40f63c65f4f49858110085b873b4028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f63c65f4f49858110085b873b4028" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32915 } } }
I20260927 22:12:52.883637 16489 sys_catalog.cc:458] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [sys.catalog]: This master's current role is: LEADER
I20260927 22:12:52.885761 16499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:12:52.903883 16499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:12:52.906199 16499 catalog_manager.cc:1259] Loaded cluster ID: 3a588df17c814a06b3b3ad7059c9f328
I20260927 22:12:52.906446 16499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:12:52.912308 16499 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:12:52.914546 16499 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028: Loaded TSK: 0
I20260927 22:12:52.948690 16499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028: Generated new TSK 1
I20260927 22:12:52.949107 16499 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:12:53.198781 16485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:12:53.199055 16485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:12:53.199193 16485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:12:53.206411 16485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:12:53.206525 16485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:12:53.206720 16485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:12:53.216529 16485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:35883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45725
--tserver_master_addrs=127.0.97.62:32915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:12:53.217803 16485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:12:53.219208 16485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:12:53.233587 16511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:53.233552 16512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:53.234720 16514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:12:53.235680 16485 server_base.cc:1034] running on GCE node
I20260927 22:12:53.236307 16485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:12:53.237308 16485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:12:53.238569 16485 hybrid_clock.cc:648] HybridClock initialized: now 1790547173238507 us; error 61 us; skew 500 ppm
I20260927 22:12:53.238950 16485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:12:53 dist-test-slave-mp03 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:12:53.242182 16485 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:12:53.244220 16485 webserver.cc:460] Webserver started at http://127.0.97.1:45725/ using document root <none> and password file <none>
I20260927 22:12:53.244912 16485 fs_manager.cc:359] Metadata directory not provided
I20260927 22:12:53.245070 16485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:12:53.250737 16485 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 22:12:53.253449 16521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:53.254923 16485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:12:53.255090 16485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "7849ea4e869a454bb71b54a3b127eb41"
format_stamp: "Formatted at 2026-09-27 22:12:30 on dist-test-slave-mp03"
server_key: "ad846b32c8ed7da503121197de244248"
server_key_iv: "b3c2e42310b0a17d97886332ed7ef086"
server_key_version: "encryptionkey@0"
I20260927 22:12:53.256042 16485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:12:53.269915 16485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:12:53.274505 16485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:12:53.274837 16485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:12:53.275923 16485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:12:53.277432 16485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:12:53.277516 16485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:53.277639 16485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:12:53.277740 16485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:53.313498 16485 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35883
I20260927 22:12:53.313591 16633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35883 every 8 connection(s)
I20260927 22:12:53.315117 16485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:12:53.319581 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 16485
I20260927 22:12:53.320361 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33355
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:32915
--webserver_port=45859
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:12:53 dist-test-slave-mp03 krb5kdc[15862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:12:53.343132 16637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:53.318093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.1:55479)
0927 22:12:53.318959 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:53.319028 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:53.321730 (+ 2702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:53.323225 (+ 1495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:53.323239 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:53.324040 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:12:53.324816 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:53.324844 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:53.326503 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:53.326511 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:53.328401 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:53.328413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:53.329926 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:53.329932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:53.330136 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:53.330908 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:12:53.330933 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:12:53.333301 (+ 2368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:12:53.338742 (+ 5441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:53.338749 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:53.338762 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:53.339192 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:53.339715 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:53.339719 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:53.339721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:53.339889 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:53.340852 (+ 963us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:12:53.340858 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:12:53.341247 (+ 389us) client_negotiation.cc:769] Sending connection context
0927 22:12:53.341478 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 22:12:53.341876 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":342,"threads_started":1}
I20260927 22:12:53.343833 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:53.317948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55479 (local address 127.0.97.62:32915)
0927 22:12:53.318690 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:53.318707 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:53.319186 (+ 479us) server_negotiation.cc:407] Connection header received
0927 22:12:53.322301 (+ 3115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:53.322313 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:53.322716 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:53.323025 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:53.325047 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:53.326328 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:53.328549 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:53.329678 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:53.329871 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:53.333651 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:53.333667 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:53.333693 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:53.333754 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:53.338561 (+ 4807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:53.339352 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:53.339363 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:53.339374 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:53.339599 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:53.340016 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:53.340023 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:53.340029 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:53.340721 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:53.340946 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:53.341795 (+ 849us) server_negotiation.cc:299] Negotiation successful
0927 22:12:53.342306 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":284,"threads_started":1}
I20260927 22:12:53.346865 16634 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32915
I20260927 22:12:53.347268 16634 heartbeater.cc:464] Registering TS with master...
I20260927 22:12:53.348269 16634 heartbeater.cc:511] Master 127.0.97.62:32915 requested a full tablet report, sending...
I20260927 22:12:53.350819 16448 ts_manager.cc:194] Registered new tserver with Master: 7849ea4e869a454bb71b54a3b127eb41 (127.0.97.1:35883)
I20260927 22:12:53.352346 16448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55479
W20260927 22:12:53.662364 16638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:12:53.662735 16638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:12:53.662923 16638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:12:53.670735 16638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:12:53.670863 16638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:12:53.671026 16638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:12:53.681159 16638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33355
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45859
--tserver_master_addrs=127.0.97.62:32915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:12:53.682441 16638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:12:53.683776 16638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:12:53.697177 16643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:53.697244 16646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:53.697212 16644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:12:53.698370 16638 server_base.cc:1034] running on GCE node
I20260927 22:12:53.699038 16638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:12:53.700042 16638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:12:53.701287 16638 hybrid_clock.cc:648] HybridClock initialized: now 1790547173701218 us; error 65 us; skew 500 ppm
I20260927 22:12:53.701634 16638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:12:53 dist-test-slave-mp03 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:12:53.704923 16638 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:12:53.706493 16638 webserver.cc:460] Webserver started at http://127.0.97.2:45859/ using document root <none> and password file <none>
I20260927 22:12:53.707271 16638 fs_manager.cc:359] Metadata directory not provided
I20260927 22:12:53.707433 16638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:12:53.712960 16638 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:12:53.715853 16653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:53.717332 16638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:12:53.717516 16638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "6a7bf25b4fc5470184bc2f1f7c530311"
format_stamp: "Formatted at 2026-09-27 22:12:30 on dist-test-slave-mp03"
server_key: "380b904cf602a8d612b1a3403aaa1ea1"
server_key_iv: "8eb7b6471c652a9281a92536eaa61a7f"
server_key_version: "encryptionkey@0"
I20260927 22:12:53.718542 16638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:12:53.737264 16638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:12:53.741658 16638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:12:53.741943 16638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:12:53.743023 16638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:12:53.744341 16638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:12:53.744457 16638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:53.744594 16638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:12:53.744679 16638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:53.779247 16638 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33355
I20260927 22:12:53.779332 16765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33355 every 8 connection(s)
I20260927 22:12:53.780808 16638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:12:53.788358 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 16638
I20260927 22:12:53.788960 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:45843
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:32915
--webserver_port=33819
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:12:53 dist-test-slave-mp03 krb5kdc[15862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:12:53.802737 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:53.783497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40361 (local address 127.0.97.62:32915)
0927 22:12:53.783690 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:53.783694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:53.784748 (+ 1054us) server_negotiation.cc:407] Connection header received
0927 22:12:53.786754 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:53.786761 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:53.786870 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:53.786998 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:53.788973 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:53.789746 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:53.791401 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:53.791839 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:53.791988 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:53.795649 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:53.795656 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:53.795665 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:53.795718 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:53.799267 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:53.800122 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:53.800130 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:53.800137 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:53.800285 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:53.800717 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:53.800725 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:53.800731 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:53.801138 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:53.801291 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:53.801977 (+ 686us) server_negotiation.cc:299] Negotiation successful
0927 22:12:53.802248 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:12:53.803548 16768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:53.783826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.2:40361)
0927 22:12:53.784577 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:53.784627 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:53.786388 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:53.787419 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:53.787429 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:53.788079 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:12:53.788775 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:53.788795 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:53.789907 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:53.789918 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:53.791251 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:53.791263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:53.792082 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:53.792089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:53.792338 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:53.793050 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:12:53.793073 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:12:53.795390 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:12:53.799459 (+ 4069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:53.799467 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:53.799482 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:53.799975 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:53.800410 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:53.800414 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:53.800416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:53.800599 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:53.801256 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:12:53.801262 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:12:53.801599 (+ 337us) client_negotiation.cc:769] Sending connection context
0927 22:12:53.801831 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 22:12:53.802244 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":288,"threads_started":1}
I20260927 22:12:53.805239 16766 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32915
I20260927 22:12:53.805588 16766 heartbeater.cc:464] Registering TS with master...
I20260927 22:12:53.806339 16766 heartbeater.cc:511] Master 127.0.97.62:32915 requested a full tablet report, sending...
I20260927 22:12:53.807920 16448 ts_manager.cc:194] Registered new tserver with Master: 6a7bf25b4fc5470184bc2f1f7c530311 (127.0.97.2:33355)
I20260927 22:12:53.808774 16448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40361
W20260927 22:12:54.139101 16769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:12:54.139418 16769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:12:54.139521 16769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:12:54.146912 16769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:12:54.147037 16769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:12:54.147164 16769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:12:54.157327 16769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:45843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=33819
--tserver_master_addrs=127.0.97.62:32915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:12:54.158756 16769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:12:54.160192 16769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:12:54.174325 16774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:54.174332 16775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:12:54.174631 16777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:12:54.176002 16769 server_base.cc:1034] running on GCE node
I20260927 22:12:54.176617 16769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:12:54.177735 16769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:12:54.178994 16769 hybrid_clock.cc:648] HybridClock initialized: now 1790547174178925 us; error 62 us; skew 500 ppm
I20260927 22:12:54.179356 16769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:12:54 dist-test-slave-mp03 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:12:54.182839 16769 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:12:54.184409 16769 webserver.cc:460] Webserver started at http://127.0.97.3:33819/ using document root <none> and password file <none>
I20260927 22:12:54.185156 16769 fs_manager.cc:359] Metadata directory not provided
I20260927 22:12:54.185322 16769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:12:54.191016 16769 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 22:12:54.193691 16784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:54.195163 16769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:12:54.195302 16769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "b682cc8bf46e4d209666aced872365b0"
format_stamp: "Formatted at 2026-09-27 22:12:31 on dist-test-slave-mp03"
server_key: "4474b14bb30c9e152a8c674b090159d4"
server_key_iv: "9e3ec151881c1d199a6bd93c73f082f0"
server_key_version: "encryptionkey@0"
I20260927 22:12:54.196401 16769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:12:54.210566 16769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:12:54.214803 16769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:12:54.215052 16769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:12:54.216169 16769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:12:54.217486 16769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:12:54.217586 16769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:54.217669 16769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:12:54.217730 16769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:12:54.251945 16769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45843
I20260927 22:12:54.252031 16896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45843 every 8 connection(s)
I20260927 22:12:54.253461 16769 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:12:54.256955 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 16769
Sep 27 22:12:54 dist-test-slave-mp03 krb5kdc[15862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:12:54.273397 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:54.257808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46168 (local address 127.0.97.62:32915)
0927 22:12:54.258274 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:54.258280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:54.258710 (+ 430us) server_negotiation.cc:407] Connection header received
0927 22:12:54.258956 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:54.258961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:54.259045 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:54.259162 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:54.260871 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.261973 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.263475 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.264108 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.264239 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:54.267001 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:54.267007 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:54.267012 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:54.267053 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:54.270232 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:54.271050 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:54.271057 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:54.271066 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:54.271216 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:54.271709 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:54.271716 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:54.271722 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:54.272122 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:54.272405 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:54.272637 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 22:12:54.272897 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":251,"threads_started":1}
I20260927 22:12:54.276527 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:54.256040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46675 (local address 127.0.97.62:32915)
0927 22:12:54.256257 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:54.256264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:54.257643 (+ 1379us) server_negotiation.cc:407] Connection header received
0927 22:12:54.260137 (+ 2494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:54.260142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:54.260217 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:54.260331 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:54.262202 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.263114 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.265219 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.265641 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.265771 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:54.269699 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:54.269707 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:54.269712 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:54.269768 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:54.273134 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:54.273944 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:54.273952 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:54.273961 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:54.274096 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:54.274647 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:54.274658 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:54.274660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:54.274966 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:54.275122 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:54.275965 (+ 843us) server_negotiation.cc:299] Negotiation successful
0927 22:12:54.276172 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:12:54.277521 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:54.256635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.3:46675)
0927 22:12:54.257465 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:54.257512 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:54.259826 (+ 2314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:54.260585 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:54.260594 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:54.261303 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:12:54.262063 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:54.262090 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.263268 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.263274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:54.265067 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:54.265079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.265866 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.265875 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:54.266126 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:54.266813 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:12:54.266835 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:12:54.269491 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:12:54.273254 (+ 3763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:54.273261 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:54.273273 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:54.273805 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:54.274224 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:12:54.274229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:12:54.274233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:12:54.274481 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:12:54.275261 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:12:54.275267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:12:54.275592 (+ 325us) client_negotiation.cc:769] Sending connection context
0927 22:12:54.275869 (+ 277us) client_negotiation.cc:240] Negotiation successful
0927 22:12:54.276426 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":367,"threads_started":1}
I20260927 22:12:54.278903 16897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32915
I20260927 22:12:54.279233 16897 heartbeater.cc:464] Registering TS with master...
I20260927 22:12:54.279935 16897 heartbeater.cc:511] Master 127.0.97.62:32915 requested a full tablet report, sending...
I20260927 22:12:54.281327 16448 ts_manager.cc:194] Registered new tserver with Master: b682cc8bf46e4d209666aced872365b0 (127.0.97.3:45843)
I20260927 22:12:54.282131 16448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46675
I20260927 22:12:54.288450 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:12:54.295252 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:54.289305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46180 (local address 127.0.97.62:32915)
0927 22:12:54.289507 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:54.289511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:54.290011 (+ 500us) server_negotiation.cc:407] Connection header received
0927 22:12:54.290203 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:54.290207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:54.290337 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:54.290480 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:12:54.290939 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.291708 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.293307 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.293779 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.293879 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:54.294381 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:12:54.294700 (+ 319us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 22:12:54.295045 (+ 345us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46180: token expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 22:12:54.295408 16636 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46180: token expired
W20260927 22:12:54.295892 16370 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:32915)
I20260927 22:12:54.308593 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:54.296512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.0.97.62:32915)
0927 22:12:54.296699 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:54.296703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:54.296720 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:12:54.296760 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:54.296764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:54.296823 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:54.296920 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:54.298245 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.299083 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.300379 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.300767 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.300861 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:54.303466 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:54.303470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:54.303474 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:54.303513 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:54.306198 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:54.306927 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:54.306930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:54.306932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:54.307068 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:54.307427 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:54.307430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:54.307432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:54.307748 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:54.307926 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:54.308181 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 22:12:54.308339 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 22:12:54.312162 16448 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:12:54.322111 16448 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:12:54.345705 16910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:54.335176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40310 (local address 127.0.97.3:45843)
0927 22:12:54.337677 (+ 2501us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:54.337685 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:54.337712 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:12:54.337802 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:54.337810 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:54.338023 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:54.338232 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:12:54.338975 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.340421 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.343355 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.344505 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.344654 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:54.344909 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:54.345184 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 22:12:54.345345 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2331,"thread_start_us":2128,"threads_started":1}
I20260927 22:12:54.345803 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:54.334788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45843 (local address 127.0.0.1:40310)
0927 22:12:54.335856 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:54.335873 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:54.336008 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:54.338234 (+ 2226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:54.338239 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:54.338265 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:12:54.338793 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:54.338804 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.340588 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.340593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:54.342933 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:54.342948 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.344724 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.344730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:54.344924 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:54.344980 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 22:12:54.345139 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 22:12:54.345339 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"mutex_wait_us":255,"thread_start_us":997,"threads_started":1}
I20260927 22:12:54.346138 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:54.334145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33355 (local address 127.0.0.1:55880)
0927 22:12:54.335130 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:54.335179 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:54.335335 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:54.337735 (+ 2400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:54.337744 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:54.337777 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:12:54.338279 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:54.338298 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.340615 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.340621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:54.343351 (+ 2730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:54.343366 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.345278 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.345284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:54.345472 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:54.345522 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 22:12:54.345591 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 22:12:54.345724 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":126,"thread_start_us":792,"threads_started":1}
I20260927 22:12:54.346771 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:54.336016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57748 (local address 127.0.97.1:35883)
0927 22:12:54.337374 (+ 1358us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:54.337382 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:54.337419 (+ 37us) server_negotiation.cc:407] Connection header received
0927 22:12:54.337505 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:54.337512 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:54.337719 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:54.337896 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:12:54.338829 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.340359 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.343710 (+ 3351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.345062 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.345203 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:54.345430 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:54.346227 (+ 797us) server_negotiation.cc:299] Negotiation successful
0927 22:12:54.346408 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1204,"thread_start_us":1080,"threads_started":1}
I20260927 22:12:54.347158 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:54.334367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55880 (local address 127.0.97.2:33355)
0927 22:12:54.337100 (+ 2733us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:54.337111 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:54.337141 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:12:54.337249 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:54.337257 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:54.337508 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:54.337705 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:12:54.338525 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.340442 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.343560 (+ 3118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.345012 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.346373 (+ 1361us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:54.346562 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:54.346674 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 22:12:54.346851 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2557,"thread_start_us":2448,"threads_started":1}
I20260927 22:12:54.347193 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:54.335679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35883 (local address 127.0.0.1:57748)
0927 22:12:54.336534 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:54.336549 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:54.336661 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:54.337996 (+ 1335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:54.338001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:54.338027 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:12:54.338643 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:54.338659 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.340507 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.340513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:54.343549 (+ 3036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:54.343560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.345584 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.345590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:54.345755 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:54.345782 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 22:12:54.345876 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 22:12:54.346005 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"mutex_wait_us":370,"thread_start_us":590,"threads_started":1}
I20260927 22:12:54.350147 16832 tablet_service.cc:1467] Processing CreateTablet for tablet b94bac91f56d4be0b9d2f497a9bd8899 (DEFAULT_TABLE table=test-table [id=3a5e89a4c7c64919956aaa631b4e13ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:12:54.350694 16701 tablet_service.cc:1467] Processing CreateTablet for tablet b94bac91f56d4be0b9d2f497a9bd8899 (DEFAULT_TABLE table=test-table [id=3a5e89a4c7c64919956aaa631b4e13ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:12:54.350694 16569 tablet_service.cc:1467] Processing CreateTablet for tablet b94bac91f56d4be0b9d2f497a9bd8899 (DEFAULT_TABLE table=test-table [id=3a5e89a4c7c64919956aaa631b4e13ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:12:54.351454 16832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b94bac91f56d4be0b9d2f497a9bd8899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:12:54.351969 16569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b94bac91f56d4be0b9d2f497a9bd8899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:12:54.351986 16701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b94bac91f56d4be0b9d2f497a9bd8899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:12:54.355904 16634 heartbeater.cc:503] Master 127.0.97.62:32915 was elected leader, sending a full tablet report...
I20260927 22:12:54.364158 16913 tablet_bootstrap.cc:492] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0: Bootstrap starting.
I20260927 22:12:54.367689 16914 tablet_bootstrap.cc:492] T b94bac91f56d4be0b9d2f497a9bd8899 P 7849ea4e869a454bb71b54a3b127eb41: Bootstrap starting.
I20260927 22:12:54.369153 16915 tablet_bootstrap.cc:492] T b94bac91f56d4be0b9d2f497a9bd8899 P 6a7bf25b4fc5470184bc2f1f7c530311: Bootstrap starting.
I20260927 22:12:54.369315 16913 tablet_bootstrap.cc:654] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0: Neither blocks nor log segments found. Creating new log.
I20260927 22:12:54.371150 16913 log.cc:824] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0: Log is configured to *not* fsync() on all Append() calls
I20260927 22:12:54.372483 16914 tablet_bootstrap.cc:654] T b94bac91f56d4be0b9d2f497a9bd8899 P 7849ea4e869a454bb71b54a3b127eb41: Neither blocks nor log segments found. Creating new log.
I20260927 22:12:54.373481 16915 tablet_bootstrap.cc:654] T b94bac91f56d4be0b9d2f497a9bd8899 P 6a7bf25b4fc5470184bc2f1f7c530311: Neither blocks nor log segments found. Creating new log.
I20260927 22:12:54.374166 16914 log.cc:824] T b94bac91f56d4be0b9d2f497a9bd8899 P 7849ea4e869a454bb71b54a3b127eb41: Log is configured to *not* fsync() on all Append() calls
I20260927 22:12:54.374976 16915 log.cc:824] T b94bac91f56d4be0b9d2f497a9bd8899 P 6a7bf25b4fc5470184bc2f1f7c530311: Log is configured to *not* fsync() on all Append() calls
I20260927 22:12:54.382287 16913 tablet_bootstrap.cc:492] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0: No bootstrap required, opened a new log
I20260927 22:12:54.382695 16913 ts_tablet_manager.cc:1397] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.008s
I20260927 22:12:54.383826 16915 tablet_bootstrap.cc:492] T b94bac91f56d4be0b9d2f497a9bd8899 P 6a7bf25b4fc5470184bc2f1f7c530311: No bootstrap required, opened a new log
I20260927 22:12:54.384104 16915 ts_tablet_manager.cc:1397] T b94bac91f56d4be0b9d2f497a9bd8899 P 6a7bf25b4fc5470184bc2f1f7c530311: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260927 22:12:54.389365 16914 tablet_bootstrap.cc:492] T b94bac91f56d4be0b9d2f497a9bd8899 P 7849ea4e869a454bb71b54a3b127eb41: No bootstrap required, opened a new log
I20260927 22:12:54.389642 16914 ts_tablet_manager.cc:1397] T b94bac91f56d4be0b9d2f497a9bd8899 P 7849ea4e869a454bb71b54a3b127eb41: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.006s
I20260927 22:12:54.391593 16915 raft_consensus.cc:348] T b94bac91f56d4be0b9d2f497a9bd8899 P 6a7bf25b4fc5470184bc2f1f7c530311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7bf25b4fc5470184bc2f1f7c530311" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } } peers { permanent_uuid: "7849ea4e869a454bb71b54a3b127eb41" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35883 } } peers { permanent_uuid: "b682cc8bf46e4d209666aced872365b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45843 } }
I20260927 22:12:54.391984 16915 raft_consensus.cc:374] T b94bac91f56d4be0b9d2f497a9bd8899 P 6a7bf25b4fc5470184bc2f1f7c530311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:12:54.392083 16915 raft_consensus.cc:729] T b94bac91f56d4be0b9d2f497a9bd8899 P 6a7bf25b4fc5470184bc2f1f7c530311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a7bf25b4fc5470184bc2f1f7c530311, State: Initialized, Role: FOLLOWER
I20260927 22:12:54.392771 16915 consensus_queue.cc:260] T b94bac91f56d4be0b9d2f497a9bd8899 P 6a7bf25b4fc5470184bc2f1f7c530311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7bf25b4fc5470184bc2f1f7c530311" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } } peers { permanent_uuid: "7849ea4e869a454bb71b54a3b127eb41" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35883 } } peers { permanent_uuid: "b682cc8bf46e4d209666aced872365b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45843 } }
I20260927 22:12:54.395350 16766 heartbeater.cc:503] Master 127.0.97.62:32915 was elected leader, sending a full tablet report...
I20260927 22:12:54.395256 16913 raft_consensus.cc:348] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7bf25b4fc5470184bc2f1f7c530311" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } } peers { permanent_uuid: "7849ea4e869a454bb71b54a3b127eb41" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35883 } } peers { permanent_uuid: "b682cc8bf46e4d209666aced872365b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45843 } }
I20260927 22:12:54.395687 16913 raft_consensus.cc:374] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:12:54.395768 16913 raft_consensus.cc:729] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b682cc8bf46e4d209666aced872365b0, State: Initialized, Role: FOLLOWER
I20260927 22:12:54.396075 16915 ts_tablet_manager.cc:1428] T b94bac91f56d4be0b9d2f497a9bd8899 P 6a7bf25b4fc5470184bc2f1f7c530311: Time spent starting tablet: real 0.012s user 0.006s sys 0.007s
I20260927 22:12:54.396452 16913 consensus_queue.cc:260] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7bf25b4fc5470184bc2f1f7c530311" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } } peers { permanent_uuid: "7849ea4e869a454bb71b54a3b127eb41" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35883 } } peers { permanent_uuid: "b682cc8bf46e4d209666aced872365b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45843 } }
I20260927 22:12:54.399029 16897 heartbeater.cc:503] Master 127.0.97.62:32915 was elected leader, sending a full tablet report...
I20260927 22:12:54.399662 16913 ts_tablet_manager.cc:1428] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0: Time spent starting tablet: real 0.017s user 0.010s sys 0.005s
I20260927 22:12:54.400759 16914 raft_consensus.cc:348] T b94bac91f56d4be0b9d2f497a9bd8899 P 7849ea4e869a454bb71b54a3b127eb41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7bf25b4fc5470184bc2f1f7c530311" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } } peers { permanent_uuid: "7849ea4e869a454bb71b54a3b127eb41" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35883 } } peers { permanent_uuid: "b682cc8bf46e4d209666aced872365b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45843 } }
I20260927 22:12:54.401270 16914 raft_consensus.cc:374] T b94bac91f56d4be0b9d2f497a9bd8899 P 7849ea4e869a454bb71b54a3b127eb41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:12:54.401386 16914 raft_consensus.cc:729] T b94bac91f56d4be0b9d2f497a9bd8899 P 7849ea4e869a454bb71b54a3b127eb41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7849ea4e869a454bb71b54a3b127eb41, State: Initialized, Role: FOLLOWER
I20260927 22:12:54.402103 16914 consensus_queue.cc:260] T b94bac91f56d4be0b9d2f497a9bd8899 P 7849ea4e869a454bb71b54a3b127eb41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7bf25b4fc5470184bc2f1f7c530311" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } } peers { permanent_uuid: "7849ea4e869a454bb71b54a3b127eb41" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35883 } } peers { permanent_uuid: "b682cc8bf46e4d209666aced872365b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45843 } }
I20260927 22:12:54.403950 16914 ts_tablet_manager.cc:1428] T b94bac91f56d4be0b9d2f497a9bd8899 P 7849ea4e869a454bb71b54a3b127eb41: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260927 22:12:54.463963 16920 raft_consensus.cc:482] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:12:54.464296 16920 raft_consensus.cc:504] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7bf25b4fc5470184bc2f1f7c530311" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } } peers { permanent_uuid: "7849ea4e869a454bb71b54a3b127eb41" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35883 } } peers { permanent_uuid: "b682cc8bf46e4d209666aced872365b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45843 } }
I20260927 22:12:54.466086 16920 leader_election.cc:288] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a7bf25b4fc5470184bc2f1f7c530311 (127.0.97.2:33355), 7849ea4e869a454bb71b54a3b127eb41 (127.0.97.1:35883)
I20260927 22:12:54.472568 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:54.466443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33355 (local address 127.0.97.3:47081)
0927 22:12:54.466689 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:54.466714 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:54.466863 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:54.467297 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:54.467303 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:54.467328 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:12:54.467555 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:54.467569 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.468748 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.468752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:54.470703 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:54.470720 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.471981 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.471988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:54.472135 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:54.472161 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 22:12:54.472244 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 22:12:54.472335 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 22:12:54.472664 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:54.466526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47081 (local address 127.0.97.2:33355)
0927 22:12:54.466742 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:54.466746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:54.466773 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:12:54.467073 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:54.467079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:54.467160 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:54.467267 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:12:54.467718 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.468609 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.470855 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.471814 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.471946 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:54.472057 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:54.472287 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 22:12:54.472418 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 22:12:54.473881 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:54.467149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47639 (local address 127.0.97.1:35883)
0927 22:12:54.467549 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:54.467554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:54.467748 (+ 194us) server_negotiation.cc:407] Connection header received
0927 22:12:54.467882 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:54.467887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:54.467960 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:54.468051 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:12:54.468633 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.469378 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.471809 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.472959 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.473097 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:54.473204 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:54.473611 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 22:12:54.473713 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257}
I20260927 22:12:54.473692 16721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b94bac91f56d4be0b9d2f497a9bd8899" candidate_uuid: "b682cc8bf46e4d209666aced872365b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a7bf25b4fc5470184bc2f1f7c530311" is_pre_election: true
I20260927 22:12:54.474121 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:54.466935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35883 (local address 127.0.97.3:47639)
0927 22:12:54.467667 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0927 22:12:54.467682 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:12:54.467787 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:12:54.468077 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:12:54.468080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:12:54.468099 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:12:54.468480 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:54.468489 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.469530 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.469536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:54.471551 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:12:54.471566 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.473180 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.473190 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:12:54.473409 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:54.473446 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:12:54.473557 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 22:12:54.473690 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":1,"thread_start_us":380,"threads_started":1}
I20260927 22:12:54.474294 16721 raft_consensus.cc:2454] T b94bac91f56d4be0b9d2f497a9bd8899 P 6a7bf25b4fc5470184bc2f1f7c530311 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b682cc8bf46e4d209666aced872365b0 in term 0.
I20260927 22:12:54.475063 16787 leader_election.cc:302] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a7bf25b4fc5470184bc2f1f7c530311, b682cc8bf46e4d209666aced872365b0; no voters:
I20260927 22:12:54.475227 16589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b94bac91f56d4be0b9d2f497a9bd8899" candidate_uuid: "b682cc8bf46e4d209666aced872365b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7849ea4e869a454bb71b54a3b127eb41" is_pre_election: true
I20260927 22:12:54.475494 16920 raft_consensus.cc:2781] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:12:54.475679 16920 raft_consensus.cc:482] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:12:54.475813 16920 raft_consensus.cc:3037] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:12:54.475776 16589 raft_consensus.cc:2454] T b94bac91f56d4be0b9d2f497a9bd8899 P 7849ea4e869a454bb71b54a3b127eb41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b682cc8bf46e4d209666aced872365b0 in term 0.
I20260927 22:12:54.478503 16920 raft_consensus.cc:504] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7bf25b4fc5470184bc2f1f7c530311" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } } peers { permanent_uuid: "7849ea4e869a454bb71b54a3b127eb41" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35883 } } peers { permanent_uuid: "b682cc8bf46e4d209666aced872365b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45843 } }
I20260927 22:12:54.479295 16920 leader_election.cc:288] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0 [CANDIDATE]: Term 1 election: Requested vote from peers 6a7bf25b4fc5470184bc2f1f7c530311 (127.0.97.2:33355), 7849ea4e869a454bb71b54a3b127eb41 (127.0.97.1:35883)
I20260927 22:12:54.479846 16589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b94bac91f56d4be0b9d2f497a9bd8899" candidate_uuid: "b682cc8bf46e4d209666aced872365b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7849ea4e869a454bb71b54a3b127eb41"
I20260927 22:12:54.479838 16721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b94bac91f56d4be0b9d2f497a9bd8899" candidate_uuid: "b682cc8bf46e4d209666aced872365b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a7bf25b4fc5470184bc2f1f7c530311"
I20260927 22:12:54.480093 16721 raft_consensus.cc:3037] T b94bac91f56d4be0b9d2f497a9bd8899 P 6a7bf25b4fc5470184bc2f1f7c530311 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:12:54.480093 16589 raft_consensus.cc:3037] T b94bac91f56d4be0b9d2f497a9bd8899 P 7849ea4e869a454bb71b54a3b127eb41 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:12:54.483584 16589 raft_consensus.cc:2454] T b94bac91f56d4be0b9d2f497a9bd8899 P 7849ea4e869a454bb71b54a3b127eb41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b682cc8bf46e4d209666aced872365b0 in term 1.
I20260927 22:12:54.483584 16721 raft_consensus.cc:2454] T b94bac91f56d4be0b9d2f497a9bd8899 P 6a7bf25b4fc5470184bc2f1f7c530311 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b682cc8bf46e4d209666aced872365b0 in term 1.
I20260927 22:12:54.484019 16787 leader_election.cc:302] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a7bf25b4fc5470184bc2f1f7c530311, b682cc8bf46e4d209666aced872365b0; no voters:
I20260927 22:12:54.484318 16920 raft_consensus.cc:2781] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:12:54.484788 16920 raft_consensus.cc:686] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0 [term 1 LEADER]: Becoming Leader. State: Replica: b682cc8bf46e4d209666aced872365b0, State: Running, Role: LEADER
I20260927 22:12:54.485409 16920 consensus_queue.cc:237] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7bf25b4fc5470184bc2f1f7c530311" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } } peers { permanent_uuid: "7849ea4e869a454bb71b54a3b127eb41" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35883 } } peers { permanent_uuid: "b682cc8bf46e4d209666aced872365b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45843 } }
I20260927 22:12:54.490257 16447 catalog_manager.cc:5495] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b682cc8bf46e4d209666aced872365b0 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b682cc8bf46e4d209666aced872365b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7bf25b4fc5470184bc2f1f7c530311" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7849ea4e869a454bb71b54a3b127eb41" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b682cc8bf46e4d209666aced872365b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45843 } health_report { overall_health: HEALTHY } } }
W20260927 22:12:54.505115 16898 tablet.cc:2369] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:12:54.513510 16910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:54.508704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.0.97.3:45843)
0927 22:12:54.508890 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:54.508904 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:54.508923 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:12:54.509012 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:54.509016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:54.509094 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:54.509232 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:12:54.509537 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.510244 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.511529 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.511914 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.512014 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:54.512518 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:12:54.513046 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:12:54.513183 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:54.513224 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 22:12:54.513288 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:12:54.521854 16721 raft_consensus.cc:1261] T b94bac91f56d4be0b9d2f497a9bd8899 P 6a7bf25b4fc5470184bc2f1f7c530311 [term 1 FOLLOWER]: Refusing update from remote peer b682cc8bf46e4d209666aced872365b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:12:54.521852 16589 raft_consensus.cc:1261] T b94bac91f56d4be0b9d2f497a9bd8899 P 7849ea4e869a454bb71b54a3b127eb41 [term 1 FOLLOWER]: Refusing update from remote peer b682cc8bf46e4d209666aced872365b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:12:54.522986 16923 consensus_queue.cc:1035] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a7bf25b4fc5470184bc2f1f7c530311" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:12:54.523336 16920 consensus_queue.cc:1035] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7849ea4e869a454bb71b54a3b127eb41" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:12:54.532713 16767 tablet.cc:2369] T b94bac91f56d4be0b9d2f497a9bd8899 P 6a7bf25b4fc5470184bc2f1f7c530311: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:12:54.537822 16928 mvcc.cc:205] Tried to move back new op lower bound from 7334081226830618624 to 7334081226697142272. Current Snapshot: MvccSnapshot[applied={T|T < 7334081226830618624}]
I20260927 22:12:54.538427 16926 mvcc.cc:205] Tried to move back new op lower bound from 7334081226830618624 to 7334081226697142272. Current Snapshot: MvccSnapshot[applied={T|T < 7334081226830618624}]
I20260927 22:12:54.551771 16448 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:12:54.552089 16448 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:12:54.555914 16448 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028: Sending DeleteTablet for 3 replicas of tablet b94bac91f56d4be0b9d2f497a9bd8899
I20260927 22:12:54.557297 16569 tablet_service.cc:1514] Processing DeleteTablet for tablet b94bac91f56d4be0b9d2f497a9bd8899 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:12:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57748
I20260927 22:12:54.557297 16701 tablet_service.cc:1514] Processing DeleteTablet for tablet b94bac91f56d4be0b9d2f497a9bd8899 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:12:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55880
I20260927 22:12:54.558012 16832 tablet_service.cc:1514] Processing DeleteTablet for tablet b94bac91f56d4be0b9d2f497a9bd8899 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:12:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40310
I20260927 22:12:54.558480 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 16485
I20260927 22:12:54.559480 16936 tablet_replica.cc:330] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0: stopping tablet replica
I20260927 22:12:54.559592 16935 tablet_replica.cc:330] T b94bac91f56d4be0b9d2f497a9bd8899 P 7849ea4e869a454bb71b54a3b127eb41: stopping tablet replica
I20260927 22:12:54.559959 16936 raft_consensus.cc:2229] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:12:54.560050 16935 raft_consensus.cc:2229] T b94bac91f56d4be0b9d2f497a9bd8899 P 7849ea4e869a454bb71b54a3b127eb41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:12:54.560649 16935 raft_consensus.cc:2258] T b94bac91f56d4be0b9d2f497a9bd8899 P 7849ea4e869a454bb71b54a3b127eb41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:12:54.560719 16936 raft_consensus.cc:2258] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:12:54.561491 16937 tablet_replica.cc:330] T b94bac91f56d4be0b9d2f497a9bd8899 P 6a7bf25b4fc5470184bc2f1f7c530311: stopping tablet replica
I20260927 22:12:54.561959 16937 raft_consensus.cc:2229] T b94bac91f56d4be0b9d2f497a9bd8899 P 6a7bf25b4fc5470184bc2f1f7c530311 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:12:54.562431 16937 raft_consensus.cc:2258] T b94bac91f56d4be0b9d2f497a9bd8899 P 6a7bf25b4fc5470184bc2f1f7c530311 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:12:54.563000 16936 ts_tablet_manager.cc:1905] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:12:54.564250 16935 ts_tablet_manager.cc:1905] T b94bac91f56d4be0b9d2f497a9bd8899 P 7849ea4e869a454bb71b54a3b127eb41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:12:54.570379 16936 ts_tablet_manager.cc:1918] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:12:54.570556 16936 log.cc:1196] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/b94bac91f56d4be0b9d2f497a9bd8899
I20260927 22:12:54.571146 16936 ts_tablet_manager.cc:1939] T b94bac91f56d4be0b9d2f497a9bd8899 P b682cc8bf46e4d209666aced872365b0: Deleting consensus metadata
I20260927 22:12:54.571866 16935 ts_tablet_manager.cc:1918] T b94bac91f56d4be0b9d2f497a9bd8899 P 7849ea4e869a454bb71b54a3b127eb41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:12:54.572101 16935 log.cc:1196] T b94bac91f56d4be0b9d2f497a9bd8899 P 7849ea4e869a454bb71b54a3b127eb41: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/wal/wals/b94bac91f56d4be0b9d2f497a9bd8899
I20260927 22:12:54.572686 16935 ts_tablet_manager.cc:1939] T b94bac91f56d4be0b9d2f497a9bd8899 P 7849ea4e869a454bb71b54a3b127eb41: Deleting consensus metadata
I20260927 22:12:54.573168 16433 catalog_manager.cc:4841] TS b682cc8bf46e4d209666aced872365b0 (127.0.97.3:45843): tablet b94bac91f56d4be0b9d2f497a9bd8899 (table test-table [id=3a5e89a4c7c64919956aaa631b4e13ea]) successfully deleted
I20260927 22:12:54.574635 16434 catalog_manager.cc:4841] TS 7849ea4e869a454bb71b54a3b127eb41 (127.0.97.1:35883): tablet b94bac91f56d4be0b9d2f497a9bd8899 (table test-table [id=3a5e89a4c7c64919956aaa631b4e13ea]) successfully deleted
I20260927 22:12:54.576395 16937 ts_tablet_manager.cc:1905] T b94bac91f56d4be0b9d2f497a9bd8899 P 6a7bf25b4fc5470184bc2f1f7c530311: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:12:54.580098 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:54.559800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57762 (local address 127.0.97.1:35883)
0927 22:12:54.559986 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:54.559992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:54.560014 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:12:54.560087 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:54.560094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:54.560225 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:54.560357 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:54.563066 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.564220 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.566540 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.567284 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.567429 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:54.571710 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:54.571726 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:54.571745 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:54.571793 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:54.576683 (+ 4890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:54.577550 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:54.577560 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:54.577564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:54.577732 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:54.578158 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:54.578164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:54.578168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:54.578694 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:54.579017 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:54.579411 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 22:12:54.579633 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:12:54.580904 16629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:12:54.584472 16937 ts_tablet_manager.cc:1918] T b94bac91f56d4be0b9d2f497a9bd8899 P 6a7bf25b4fc5470184bc2f1f7c530311: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:12:54.584772 16937 log.cc:1196] T b94bac91f56d4be0b9d2f497a9bd8899 P 6a7bf25b4fc5470184bc2f1f7c530311: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/b94bac91f56d4be0b9d2f497a9bd8899
I20260927 22:12:54.585376 16937 ts_tablet_manager.cc:1939] T b94bac91f56d4be0b9d2f497a9bd8899 P 6a7bf25b4fc5470184bc2f1f7c530311: Deleting consensus metadata
I20260927 22:12:54.587265 16434 catalog_manager.cc:4841] TS 6a7bf25b4fc5470184bc2f1f7c530311 (127.0.97.2:33355): tablet b94bac91f56d4be0b9d2f497a9bd8899 (table test-table [id=3a5e89a4c7c64919956aaa631b4e13ea]) successfully deleted
I20260927 22:12:54.673934 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 16485
I20260927 22:12:54.692581 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 16638
I20260927 22:12:54.712885 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:54.693536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55892 (local address 127.0.97.2:33355)
0927 22:12:54.693707 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:54.693712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:54.693730 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:12:54.693891 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:54.693895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:54.693958 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:54.694061 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:54.695845 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.696599 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.698531 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.698971 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.699083 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:54.707037 (+ 7954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:54.707052 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:54.707067 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:54.707109 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:54.709989 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:54.710843 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:54.710851 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:54.710854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:54.711028 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:54.711459 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:54.711463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:54.711465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:54.711833 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:54.712067 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:54.712402 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 22:12:54.712573 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:12:54.713690 16761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:12:54.803627 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 16638
I20260927 22:12:54.820643 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 16769
I20260927 22:12:54.835004 16910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:12:54.821405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40322 (local address 127.0.97.3:45843)
0927 22:12:54.821637 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 22:12:54.821641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:12:54.821727 (+ 86us) server_negotiation.cc:407] Connection header received
0927 22:12:54.821933 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:12:54.821937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:12:54.822001 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:12:54.822133 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:12:54.823516 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.824318 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.825679 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:12:54.826130 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:12:54.826265 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:12:54.828784 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:12:54.828794 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:12:54.828810 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:12:54.828844 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:12:54.831870 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:54.832644 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:54.832654 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:54.832663 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:54.832842 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:12:54.833263 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:12:54.833269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:12:54.833272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:12:54.833642 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:12:54.833977 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 22:12:54.834290 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 22:12:54.834531 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 22:12:54.835801 16892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:12:54.960490 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 16769
I20260927 22:12:54.981277 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 16415
I20260927 22:12:54.982548 16478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:12:55.100767 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 16415
I20260927 22:13:15.123883 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32915
--webserver_interface=127.0.97.62
--webserver_port=39915
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:13:15.466365 16942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:15.466753 16942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:15.466941 16942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:15.474298 16942 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 22:13:15.474443 16942 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 22:13:15.474526 16942 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 22:13:15.474687 16942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:13:15.474797 16942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:15.474900 16942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:15.475018 16942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:13:15.475126 16942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:13:15.485357 16942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=39915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:15.486891 16942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:15.488273 16942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:15.501157 16948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:15.501216 16947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:15.501277 16950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:15.501897 16942 server_base.cc:1034] running on GCE node
I20260927 22:13:15.502811 16942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:15.504321 16942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:15.505579 16942 hybrid_clock.cc:648] HybridClock initialized: now 1790547195505515 us; error 45 us; skew 500 ppm
I20260927 22:13:15.505971 16942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:13:15 dist-test-slave-mp03 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:13:15.516283 16942 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:13:15.517923 16942 webserver.cc:460] Webserver started at http://127.0.97.62:39915/ using document root <none> and password file <none>
I20260927 22:13:15.518735 16942 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:15.518878 16942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:15.524482 16942 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:13:15.527613 16957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:15.528973 16942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:13:15.529209 16942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "e40f63c65f4f49858110085b873b4028"
format_stamp: "Formatted at 2026-09-27 22:12:29 on dist-test-slave-mp03"
server_key: "de62939866d7bbef30bf4b5a87176dc0"
server_key_iv: "2bcc5a51a7296d0a00acdaa3bde35404"
server_key_version: "encryptionkey@0"
I20260927 22:13:15.530174 16942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:15.552287 16942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:15.556430 16942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:15.556736 16942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:15.576716 16942 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32915
I20260927 22:13:15.576772 17008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32915 every 8 connection(s)
I20260927 22:13:15.578377 16942 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:13:15.585274 17009 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:13:15.588455 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 16942
I20260927 22:13:15.588608 17009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028: Bootstrap starting.
I20260927 22:13:15.589340 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:35883
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:32915
--webserver_port=45725
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:13:15.609226 17009 log.cc:824] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:15.625926 17009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:13:15.626521 17009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028: Bootstrap complete.
I20260927 22:13:15.636474 17009 raft_consensus.cc:348] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f63c65f4f49858110085b873b4028" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32915 } }
I20260927 22:13:15.637488 17009 raft_consensus.cc:729] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e40f63c65f4f49858110085b873b4028, State: Initialized, Role: FOLLOWER
I20260927 22:13:15.638298 17009 consensus_queue.cc:260] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f63c65f4f49858110085b873b4028" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32915 } }
I20260927 22:13:15.638540 17009 raft_consensus.cc:388] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:13:15.638700 17009 raft_consensus.cc:482] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:13:15.638900 17009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [term 2 FOLLOWER]: Advancing to term 3
I20260927 22:13:15.643117 17009 raft_consensus.cc:504] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f63c65f4f49858110085b873b4028" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32915 } }
I20260927 22:13:15.643679 17009 leader_election.cc:302] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e40f63c65f4f49858110085b873b4028; no voters:
I20260927 22:13:15.644416 17009 leader_election.cc:288] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 22:13:15.644572 17013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 22:13:15.645919 17013 raft_consensus.cc:686] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [term 3 LEADER]: Becoming Leader. State: Replica: e40f63c65f4f49858110085b873b4028, State: Running, Role: LEADER
I20260927 22:13:15.646549 17013 consensus_queue.cc:237] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f63c65f4f49858110085b873b4028" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32915 } }
I20260927 22:13:15.646860 17009 sys_catalog.cc:564] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:13:15.649041 17015 sys_catalog.cc:455] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e40f63c65f4f49858110085b873b4028. Latest consensus state: current_term: 3 leader_uuid: "e40f63c65f4f49858110085b873b4028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f63c65f4f49858110085b873b4028" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32915 } } }
I20260927 22:13:15.650002 17015 sys_catalog.cc:458] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [sys.catalog]: This master's current role is: LEADER
I20260927 22:13:15.649781 17014 sys_catalog.cc:455] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e40f63c65f4f49858110085b873b4028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f63c65f4f49858110085b873b4028" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32915 } } }
I20260927 22:13:15.650161 17014 sys_catalog.cc:458] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028 [sys.catalog]: This master's current role is: LEADER
I20260927 22:13:15.656786 17021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:13:15.665848 17021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:13:15.667620 17021 catalog_manager.cc:1259] Loaded cluster ID: 3a588df17c814a06b3b3ad7059c9f328
I20260927 22:13:15.667795 17021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:13:15.672993 17021 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:13:15.674762 17021 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028: Loaded TSK: 0
I20260927 22:13:15.675010 17021 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e40f63c65f4f49858110085b873b4028: Loaded TSK: 1
F20260927 22:13:15.676373 17021 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f87e1fe99ad google::LogMessage::Fail() at ??:0
@ 0x7f87e1fec88a google::LogMessage::SendToLog() at ??:0
@ 0x7f87e1fe94de google::LogMessage::Flush() at ??:0
@ 0x7f87e1feaa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f87e5d4d7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f87e5d3c73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f87f7a1c1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f87f7a1b783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f87f7aa72a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f87f7aa70c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f87f7af15e9 std::function<>::operator()() at ??:0
@ 0x7f87f7a1fcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f87f7a1dc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f87f7aa5610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f87f7aa5439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f87e4c35345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f87e4c3891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f87e4c38799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f87e4c13eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f87e7eed6db start_thread at ??:0
@ 0x7f87e0e8361f clone at ??:0
W20260927 22:13:15.995517 17011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:15.995874 17011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:15.996066 17011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:16.003510 17011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:16.003692 17011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:16.003881 17011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:13:16.014053 17011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:35883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45725
--tserver_master_addrs=127.0.97.62:32915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:16.015419 17011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:16.016748 17011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:16.030449 17056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:16.030424 17057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:16.030421 17059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:16.031032 17011 server_base.cc:1034] running on GCE node
I20260927 22:13:16.031893 17011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:16.032893 17011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:16.034144 17011 hybrid_clock.cc:648] HybridClock initialized: now 1790547196034084 us; error 73 us; skew 500 ppm
I20260927 22:13:16.034476 17011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:13:16 dist-test-slave-mp03 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:13:16.037738 17011 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:13:16.039379 17011 webserver.cc:460] Webserver started at http://127.0.97.1:45725/ using document root <none> and password file <none>
I20260927 22:13:16.040097 17011 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:16.040249 17011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:16.045586 17011 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 22:13:16.048519 17066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:16.049963 17011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:13:16.050185 17011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "7849ea4e869a454bb71b54a3b127eb41"
format_stamp: "Formatted at 2026-09-27 22:12:30 on dist-test-slave-mp03"
server_key: "ad846b32c8ed7da503121197de244248"
server_key_iv: "b3c2e42310b0a17d97886332ed7ef086"
server_key_version: "encryptionkey@0"
I20260927 22:13:16.051288 17011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:16.073510 17011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:16.078306 17011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:16.078640 17011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:16.079689 17011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:13:16.081108 17011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:13:16.081231 17011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:16.081377 17011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:13:16.081486 17011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:16.114882 17011 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35883
I20260927 22:13:16.114938 17178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35883 every 8 connection(s)
I20260927 22:13:16.116485 17011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:13:16.121593 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:16.119655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.1:47971)
0927 22:13:16.120470 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:16.120693 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":352,"threads_started":1}
W20260927 22:13:16.122370 17179 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:32915 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:32915: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
I20260927 22:13:16.123260 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:16.122903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.1:42539)
0927 22:13:16.123089 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:16.123159 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 22:13:16.124155 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:16.123843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.1:45121)
0927 22:13:16.123962 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:16.124047 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 22:13:16.124379 17179 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:13:16.124584 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 17011
I20260927 22:13:16.125234 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33355
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:32915
--webserver_port=45859
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:13:16.455958 17182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:16.456301 17182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:16.456466 17182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:16.463733 17182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:16.463835 17182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:16.463994 17182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:13:16.473608 17182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33355
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45859
--tserver_master_addrs=127.0.97.62:32915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:16.474937 17182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:16.476260 17182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:16.495070 17188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:16.495473 17187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:16.496610 17190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:16.498634 17182 server_base.cc:1034] running on GCE node
I20260927 22:13:16.499617 17182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:16.500996 17182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:16.502225 17182 hybrid_clock.cc:648] HybridClock initialized: now 1790547196502164 us; error 59 us; skew 500 ppm
I20260927 22:13:16.502665 17182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:13:16 dist-test-slave-mp03 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:13:16.507105 17182 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:13:16.509100 17182 webserver.cc:460] Webserver started at http://127.0.97.2:45859/ using document root <none> and password file <none>
I20260927 22:13:16.510102 17182 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:16.510244 17182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:16.517998 17182 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 22:13:16.522117 17197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:16.523985 17182 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 22:13:16.524147 17182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "6a7bf25b4fc5470184bc2f1f7c530311"
format_stamp: "Formatted at 2026-09-27 22:12:30 on dist-test-slave-mp03"
server_key: "380b904cf602a8d612b1a3403aaa1ea1"
server_key_iv: "8eb7b6471c652a9281a92536eaa61a7f"
server_key_version: "encryptionkey@0"
I20260927 22:13:16.525203 17182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:16.540810 17182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:16.545311 17182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:16.545593 17182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:16.546568 17182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:13:16.548017 17182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:13:16.548110 17182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:16.548237 17182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:13:16.548313 17182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:16.581828 17182 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33355
I20260927 22:13:16.581918 17309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33355 every 8 connection(s)
I20260927 22:13:16.583356 17182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:13:16.588322 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:16.586419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.2:51001)
0927 22:13:16.587204 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:16.587393 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":338,"threads_started":1}
W20260927 22:13:16.589000 17310 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:32915 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:32915: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
I20260927 22:13:16.589275 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 17182
I20260927 22:13:16.589941 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:16.589529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.2:43389)
0927 22:13:16.589712 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:16.589800 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 22:13:16.590034 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:45843
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:32915
--webserver_port=33819
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:13:16.590932 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:16.590487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.2:34773)
0927 22:13:16.590671 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:16.590771 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 22:13:16.591189 17310 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:13:16.920535 17313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:16.920874 17313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:16.921048 17313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:16.928649 17313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:16.928807 17313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:16.928992 17313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:13:16.938972 17313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:45843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=33819
--tserver_master_addrs=127.0.97.62:32915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:16.940317 17313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:16.941793 17313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:16.954912 17319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:16.954941 17318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:16.955209 17321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:16.955976 17313 server_base.cc:1034] running on GCE node
I20260927 22:13:16.956615 17313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:16.957543 17313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:16.958768 17313 hybrid_clock.cc:648] HybridClock initialized: now 1790547196958713 us; error 71 us; skew 500 ppm
I20260927 22:13:16.959095 17313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:13:16 dist-test-slave-mp03 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:13:16.962185 17313 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:13:16.963665 17313 webserver.cc:460] Webserver started at http://127.0.97.3:33819/ using document root <none> and password file <none>
I20260927 22:13:16.964344 17313 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:16.964486 17313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:16.969676 17313 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 22:13:16.972472 17328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:16.973966 17313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:13:16.974169 17313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "b682cc8bf46e4d209666aced872365b0"
format_stamp: "Formatted at 2026-09-27 22:12:31 on dist-test-slave-mp03"
server_key: "4474b14bb30c9e152a8c674b090159d4"
server_key_iv: "9e3ec151881c1d199a6bd93c73f082f0"
server_key_version: "encryptionkey@0"
I20260927 22:13:16.975311 17313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:16.991148 17313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:16.995442 17313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:16.995750 17313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:16.996765 17313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:13:16.998132 17313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:13:16.998265 17313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:16.998400 17313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:13:16.998489 17313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:17.034780 17313 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45843
I20260927 22:13:17.034864 17440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45843 every 8 connection(s)
I20260927 22:13:17.036350 17313 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:13:17.041201 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:17.039294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.3:34305)
0927 22:13:17.040036 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:17.040224 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":299,"threads_started":1}
W20260927 22:13:17.041857 17441 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:32915 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:32915: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
I20260927 22:13:17.042738 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:17.042369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.3:47623)
0927 22:13:17.042520 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:17.042587 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 22:13:17.043637 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:17.043228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.3:36281)
0927 22:13:17.043383 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:17.043497 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 22:13:17.043704 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 17313
W20260927 22:13:17.043902 17441 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:13:17.125895 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:17.125064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.1:42977)
0927 22:13:17.125509 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:17.125591 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":209,"threads_started":1}
I20260927 22:13:17.592684 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:17.591751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.2:37943)
0927 22:13:17.592303 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:17.592421 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":222,"threads_started":1}
I20260927 22:13:18.045228 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:18.044453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.3:60683)
0927 22:13:18.044904 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:18.044991 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":217,"threads_started":1}
I20260927 22:13:18.127700 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:18.126926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.1:53635)
0927 22:13:18.127448 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:18.127534 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":291,"threads_started":1}
I20260927 22:13:18.594542 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:18.593757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.2:56001)
0927 22:13:18.594273 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:18.594368 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":266,"threads_started":1}
I20260927 22:13:19.046839 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:19.046028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.3:58159)
0927 22:13:19.046530 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:19.046675 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":286,"threads_started":1}
I20260927 22:13:19.129369 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:19.128690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.1:50529)
0927 22:13:19.129108 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:19.129196 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":210,"threads_started":1}
I20260927 22:13:19.596134 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:19.595450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.2:60953)
0927 22:13:19.595877 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:19.595955 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":231,"threads_started":1}
I20260927 22:13:20.048614 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:20.047748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.3:44733)
0927 22:13:20.048345 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:20.048436 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":372,"threads_started":1}
I20260927 22:13:20.131242 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:20.130370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.1:43899)
0927 22:13:20.130896 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:20.130992 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":298,"threads_started":1}
I20260927 22:13:20.597967 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:20.597249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.2:52335)
0927 22:13:20.597713 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:20.597794 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":232,"threads_started":1}
I20260927 22:13:21.050354 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:21.049602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.3:39123)
0927 22:13:21.050024 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:21.050156 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":207,"threads_started":1}
I20260927 22:13:21.132959 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:21.132246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.1:56185)
0927 22:13:21.132714 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:21.132794 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":247,"threads_started":1}
I20260927 22:13:21.599586 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:21.598861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.2:57429)
0927 22:13:21.599329 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:21.599406 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":228,"threads_started":1}
I20260927 22:13:22.052021 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:22.051302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.3:44585)
0927 22:13:22.051770 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:22.051852 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":239,"threads_started":1}
I20260927 22:13:22.134647 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:22.133879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.1:60349)
0927 22:13:22.134337 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:22.134457 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":225,"threads_started":1}
I20260927 22:13:22.601212 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:22.600447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.2:40615)
0927 22:13:22.600953 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:22.601043 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":291,"threads_started":1}
I20260927 22:13:23.053630 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:23.052913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.3:45875)
0927 22:13:23.053376 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:23.053468 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":239,"threads_started":1}
I20260927 22:13:23.136407 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:23.135509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.1:57665)
0927 22:13:23.136028 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:23.136165 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":234,"threads_started":1}
I20260927 22:13:23.603112 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:23.602260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.2:34005)
0927 22:13:23.602805 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:23.602882 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":216,"threads_started":1}
I20260927 22:13:24.055658 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:24.054731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.3:49381)
0927 22:13:24.055275 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:24.055389 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":244,"threads_started":1}
I20260927 22:13:24.138458 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:24.137409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.1:34867)
0927 22:13:24.138101 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:24.138249 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":437,"threads_started":1}
I20260927 22:13:24.604763 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:24.604046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.2:55471)
0927 22:13:24.604499 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:24.604579 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":248,"threads_started":1}
I20260927 22:13:25.057376 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:25.056597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.3:49121)
0927 22:13:25.057114 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:25.057210 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":289,"threads_started":1}
I20260927 22:13:25.140245 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:25.139491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.1:40483)
0927 22:13:25.139986 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:25.140074 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":246,"threads_started":1}
I20260927 22:13:25.606470 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:25.605643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.2:46217)
0927 22:13:25.606197 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:25.606292 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":299,"threads_started":1}
I20260927 22:13:26.059173 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:26.058261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.3:57295)
0927 22:13:26.058848 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:26.058939 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":277,"threads_started":1}
I20260927 22:13:26.142014 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:26.141331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.1:48529)
0927 22:13:26.141773 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:26.141855 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":244,"threads_started":1}
I20260927 22:13:26.608139 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:26.607415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.2:51521)
0927 22:13:26.607875 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:26.607960 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":244,"threads_started":1}
I20260927 22:13:27.060918 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:27.060175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.3:56601)
0927 22:13:27.060659 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:27.060746 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":273,"threads_started":1}
I20260927 22:13:27.143946 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:27.143079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.1:52735)
0927 22:13:27.143679 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:27.143770 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":359,"threads_started":1}
I20260927 22:13:27.610110 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:27.609192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.2:53519)
0927 22:13:27.609823 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:27.609925 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":312,"threads_started":1}
I20260927 22:13:28.062826 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:28.061873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.3:41227)
0927 22:13:28.062401 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:28.062536 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":274,"threads_started":1}
I20260927 22:13:28.145614 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:28.144905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.1:57213)
0927 22:13:28.145361 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:28.145460 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":250,"threads_started":1}
I20260927 22:13:28.611847 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:28.611130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.2:49933)
0927 22:13:28.611596 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:28.611677 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":239,"threads_started":1}
I20260927 22:13:29.064602 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:29.063769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.3:38669)
0927 22:13:29.064339 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:29.064426 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":288,"threads_started":1}
I20260927 22:13:29.147396 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:29.146665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.1:56977)
0927 22:13:29.147125 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:29.147211 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":214,"threads_started":1}
I20260927 22:13:29.613620 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:29.612822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.2:41603)
0927 22:13:29.613313 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:29.613403 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":243,"threads_started":1}
I20260927 22:13:30.066308 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:30.065504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.3:37649)
0927 22:13:30.065990 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:30.066107 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":224,"threads_started":1}
I20260927 22:13:30.149123 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:30.148270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.1:55983)
0927 22:13:30.148757 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:30.148863 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":257,"threads_started":1}
I20260927 22:13:30.615468 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:30.614698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.2:41429)
0927 22:13:30.615157 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:30.615241 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":228,"threads_started":1}
I20260927 22:13:31.068142 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:31.067409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.3:54685)
0927 22:13:31.067876 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:31.067960 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":232,"threads_started":1}
I20260927 22:13:31.151153 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:31.150317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.1:60589)
0927 22:13:31.150882 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:31.150969 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":325,"threads_started":1}
I20260927 22:13:31.617373 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:31.616625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.2:42959)
0927 22:13:31.617091 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:31.617176 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":233,"threads_started":1}
I20260927 22:13:32.055553 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 17011
I20260927 22:13:32.070161 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:32.069441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32915 (local address 127.0.97.3:34145)
0927 22:13:32.069899 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:32.069979 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":248,"threads_started":1}
I20260927 22:13:32.081321 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:32.057141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40766 (local address 127.0.97.1:35883)
0927 22:13:32.057694 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:32.057709 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:32.057749 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:13:32.059534 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:32.059544 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:32.059999 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:32.060307 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:32.061629 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:32.062951 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:32.064118 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:32.064748 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:32.064886 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:32.074715 (+ 9829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:32.074726 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:32.074744 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:32.074786 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:32.077989 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:32.078754 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:32.078762 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:32.078770 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:32.079001 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:32.079387 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:32.079390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:32.079393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:32.080062 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:32.080291 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:32.080680 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 22:13:32.081043 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":270,"threads_started":1}
I20260927 22:13:32.082867 17174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:32.168571 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 17011
I20260927 22:13:32.186854 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 17182
I20260927 22:13:32.204981 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:32.187886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41400 (local address 127.0.97.2:33355)
0927 22:13:32.188394 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:32.188407 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:32.188444 (+ 37us) server_negotiation.cc:407] Connection header received
0927 22:13:32.189959 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:32.189967 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:32.190412 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:32.190845 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:32.192001 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:32.193271 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:32.194514 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:32.195277 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:32.195414 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:32.197941 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:32.197948 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:32.197964 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:32.198000 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:32.201669 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:32.202422 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:32.202429 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:32.202437 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:32.202719 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:32.203120 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:32.203124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:32.203126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:32.203725 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:32.203969 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:32.204303 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 22:13:32.204684 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":253,"threads_started":1}
I20260927 22:13:32.206418 17305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:32.292949 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 17182
I20260927 22:13:32.310187 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 17313
I20260927 22:13:32.327971 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:32.311247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49232 (local address 127.0.97.3:45843)
0927 22:13:32.311832 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:32.311846 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:32.311887 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:13:32.313483 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:32.313509 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:32.313946 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:32.314235 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:32.315749 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:32.316974 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:32.318374 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:32.319089 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:32.319235 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:32.321879 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:32.321886 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:32.321902 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:32.321940 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:32.324955 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:32.325583 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:32.325589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:32.325596 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:32.325754 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:32.326078 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:32.326081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:32.326083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:32.326669 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:32.326901 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:32.327236 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 22:13:32.327675 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":292,"threads_started":1}
I20260927 22:13:32.329521 17436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:32.413378 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 17313
2026-09-27T22:13:32Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64515 ms)
[----------] 30 tests from SecurityITest (299391 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:13:32 dist-test-slave-mp03 krb5kdc[17499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:13:32 dist-test-slave-mp03 krb5kdc[17499](info): set up 2 sockets
Sep 27 22:13:32 dist-test-slave-mp03 krb5kdc[17499](info): commencing operation
krb5kdc: starting...
W20260927 22:13:33.550709 388 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:13:33 dist-test-slave-mp03 krb5kdc[17499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:13:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:13:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:13:33.793756 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35799
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:13:34.124065 17512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:34.124413 17512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:34.124588 17512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:34.131841 17512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:13:34.131985 17512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:34.132115 17512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:34.132215 17512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:13:34.132300 17512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:13:34.142253 17512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:34.143646 17512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:34.144948 17512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:34.158944 17517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:34.158980 17520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:34.159014 17518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:34.159899 17512 server_base.cc:1034] running on GCE node
I20260927 22:13:34.160809 17512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:34.162335 17512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:34.163604 17512 hybrid_clock.cc:648] HybridClock initialized: now 1790547214163532 us; error 63 us; skew 500 ppm
I20260927 22:13:34.163985 17512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:13:34 dist-test-slave-mp03 krb5kdc[17499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:13:34.167378 17512 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:13:34.168815 17512 webserver.cc:460] Webserver started at http://127.0.97.62:36305/ using document root <none> and password file <none>
I20260927 22:13:34.169677 17512 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:34.169849 17512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:34.170185 17512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:34.173282 17512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "f0ab93e0c74c40e691c3608e987f8270"
format_stamp: "Formatted at 2026-09-27 22:13:34 on dist-test-slave-mp03"
server_key: "90f441d9a7d03b3ec924abf0fe0c8c38"
server_key_iv: "2dadd33f5b6960527577b38009450678"
server_key_version: "encryptionkey@0"
I20260927 22:13:34.174050 17512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "f0ab93e0c74c40e691c3608e987f8270"
format_stamp: "Formatted at 2026-09-27 22:13:34 on dist-test-slave-mp03"
server_key: "90f441d9a7d03b3ec924abf0fe0c8c38"
server_key_iv: "2dadd33f5b6960527577b38009450678"
server_key_version: "encryptionkey@0"
I20260927 22:13:34.179234 17512 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 22:13:34.182734 17527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:34.184089 17512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 22:13:34.184257 17512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "f0ab93e0c74c40e691c3608e987f8270"
format_stamp: "Formatted at 2026-09-27 22:13:34 on dist-test-slave-mp03"
server_key: "90f441d9a7d03b3ec924abf0fe0c8c38"
server_key_iv: "2dadd33f5b6960527577b38009450678"
server_key_version: "encryptionkey@0"
I20260927 22:13:34.184487 17512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:34.212163 17512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:34.216728 17512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:34.217031 17512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:34.235355 17512 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35799
I20260927 22:13:34.235387 17578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35799 every 8 connection(s)
I20260927 22:13:34.236719 17512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:13:34.238229 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 17512
I20260927 22:13:34.238437 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:13:34.239290 388 external_mini_cluster.cc:1428] Setting key bade6bf38dfa1114e30e81dad426a612
I20260927 22:13:34.241351 17579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:34.252966 17579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0ab93e0c74c40e691c3608e987f8270: Bootstrap starting.
Sep 27 22:13:34 dist-test-slave-mp03 krb5kdc[17499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:13:34.256038 17579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0ab93e0c74c40e691c3608e987f8270: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:34.257269 17579 log.cc:824] T 00000000000000000000000000000000 P f0ab93e0c74c40e691c3608e987f8270: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:34.260188 17579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0ab93e0c74c40e691c3608e987f8270: No bootstrap required, opened a new log
I20260927 22:13:34.264488 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:34.241457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57488 (local address 127.0.97.62:35799)
0927 22:13:34.242268 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:34.242280 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:34.242323 (+ 43us) server_negotiation.cc:407] Connection header received
0927 22:13:34.244703 (+ 2380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:34.244727 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:34.245138 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:34.245573 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:34.247215 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:34.248654 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:34.250100 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:34.250956 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:34.251178 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:34.255729 (+ 4551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:34.255738 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:34.255753 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:34.255798 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:34.259305 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:34.260053 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:34.260069 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:34.260076 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:34.260257 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:34.260759 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:34.260763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:34.260766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:34.261360 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:34.261621 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:34.262166 (+ 545us) server_negotiation.cc:299] Negotiation successful
0927 22:13:34.262937 (+ 771us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":372,"threads_started":1}
I20260927 22:13:34.270085 17579 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0ab93e0c74c40e691c3608e987f8270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ab93e0c74c40e691c3608e987f8270" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35799 } }
I20260927 22:13:34.270444 17579 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0ab93e0c74c40e691c3608e987f8270 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:34.270569 17579 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0ab93e0c74c40e691c3608e987f8270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0ab93e0c74c40e691c3608e987f8270, State: Initialized, Role: FOLLOWER
I20260927 22:13:34.271255 17579 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0ab93e0c74c40e691c3608e987f8270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ab93e0c74c40e691c3608e987f8270" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35799 } }
I20260927 22:13:34.271507 17579 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0ab93e0c74c40e691c3608e987f8270 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:13:34.271652 17579 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0ab93e0c74c40e691c3608e987f8270 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:13:34.271847 17579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0ab93e0c74c40e691c3608e987f8270 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:34.273964 17579 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0ab93e0c74c40e691c3608e987f8270 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ab93e0c74c40e691c3608e987f8270" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35799 } }
I20260927 22:13:34.274456 17579 leader_election.cc:302] T 00000000000000000000000000000000 P f0ab93e0c74c40e691c3608e987f8270 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0ab93e0c74c40e691c3608e987f8270; no voters:
I20260927 22:13:34.275125 17579 leader_election.cc:288] T 00000000000000000000000000000000 P f0ab93e0c74c40e691c3608e987f8270 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:13:34.275334 17584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0ab93e0c74c40e691c3608e987f8270 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:13:34.276615 17584 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0ab93e0c74c40e691c3608e987f8270 [term 1 LEADER]: Becoming Leader. State: Replica: f0ab93e0c74c40e691c3608e987f8270, State: Running, Role: LEADER
I20260927 22:13:34.277165 17584 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0ab93e0c74c40e691c3608e987f8270 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ab93e0c74c40e691c3608e987f8270" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35799 } }
I20260927 22:13:34.277390 17579 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0ab93e0c74c40e691c3608e987f8270 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:13:34.284247 17585 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0ab93e0c74c40e691c3608e987f8270 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0ab93e0c74c40e691c3608e987f8270" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ab93e0c74c40e691c3608e987f8270" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35799 } } }
I20260927 22:13:34.284674 17585 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0ab93e0c74c40e691c3608e987f8270 [sys.catalog]: This master's current role is: LEADER
I20260927 22:13:34.285257 17586 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0ab93e0c74c40e691c3608e987f8270 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0ab93e0c74c40e691c3608e987f8270. Latest consensus state: current_term: 1 leader_uuid: "f0ab93e0c74c40e691c3608e987f8270" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ab93e0c74c40e691c3608e987f8270" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35799 } } }
I20260927 22:13:34.285576 17586 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0ab93e0c74c40e691c3608e987f8270 [sys.catalog]: This master's current role is: LEADER
W20260927 22:13:34.293366 17598 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0ab93e0c74c40e691c3608e987f8270: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:13:34.293506 17598 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:13:34.293612 17599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:13:34.295789 17599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:13:34.304505 17599 catalog_manager.cc:1347] Generated new cluster ID: 975338ace2444ee9b6b9d5b1a5a7b381
I20260927 22:13:34.304636 17599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:13:34.325455 17599 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:13:34.327168 17599 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:13:34.335764 17599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0ab93e0c74c40e691c3608e987f8270: Generated new TSK 0
I20260927 22:13:34.336650 17599 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:13:34.398540 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35799
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:13:34.741768 17606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:34.742103 17606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:34.742278 17606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:34.749683 17606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:34.749838 17606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:34.750046 17606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:13:34.760347 17606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:34.761715 17606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:34.763243 17606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:34.776476 17611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:34.776432 17612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:34.776446 17614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:34.777393 17606 server_base.cc:1034] running on GCE node
I20260927 22:13:34.778044 17606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:34.779155 17606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:34.780390 17606 hybrid_clock.cc:648] HybridClock initialized: now 1790547214780339 us; error 43 us; skew 500 ppm
I20260927 22:13:34.780737 17606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:13:34 dist-test-slave-mp03 krb5kdc[17499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:13:34.784149 17606 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:13:34.785642 17606 webserver.cc:460] Webserver started at http://127.0.97.1:39875/ using document root <none> and password file <none>
I20260927 22:13:34.786410 17606 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:34.786561 17606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:34.786906 17606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:34.790087 17606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "55632bda633d41d6932a807ec1007d00"
format_stamp: "Formatted at 2026-09-27 22:13:34 on dist-test-slave-mp03"
server_key: "2e116193fc89d7cedad3daf7b2217b1e"
server_key_iv: "a918a359ad87946360fc7e719951dc8c"
server_key_version: "encryptionkey@0"
I20260927 22:13:34.790933 17606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "55632bda633d41d6932a807ec1007d00"
format_stamp: "Formatted at 2026-09-27 22:13:34 on dist-test-slave-mp03"
server_key: "2e116193fc89d7cedad3daf7b2217b1e"
server_key_iv: "a918a359ad87946360fc7e719951dc8c"
server_key_version: "encryptionkey@0"
I20260927 22:13:34.796149 17606 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:13:34.799726 17621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:34.801482 17606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:13:34.801671 17606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "55632bda633d41d6932a807ec1007d00"
format_stamp: "Formatted at 2026-09-27 22:13:34 on dist-test-slave-mp03"
server_key: "2e116193fc89d7cedad3daf7b2217b1e"
server_key_iv: "a918a359ad87946360fc7e719951dc8c"
server_key_version: "encryptionkey@0"
I20260927 22:13:34.801909 17606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:34.817509 17606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:34.821480 17606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:34.821785 17606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:34.822891 17606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:13:34.824231 17606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:13:34.824338 17606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:34.824484 17606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:13:34.824560 17606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:34.859093 17606 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33829
I20260927 22:13:34.859146 17733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33829 every 8 connection(s)
I20260927 22:13:34.860468 17606 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:13:34.866231 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 17606
I20260927 22:13:34.866436 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:13:34.867137 388 external_mini_cluster.cc:1428] Setting key 043b4bb9d6a3fde4f0f9f0dd980b5134
Sep 27 22:13:34 dist-test-slave-mp03 krb5kdc[17499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:13:34.883471 17736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:34.863197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47659 (local address 127.0.97.62:35799)
0927 22:13:34.863704 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:34.863709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:34.864772 (+ 1063us) server_negotiation.cc:407] Connection header received
0927 22:13:34.866961 (+ 2189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:34.866966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:34.867079 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:34.867202 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:34.869206 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:34.870271 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:34.871998 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:34.872702 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:34.872842 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:34.876565 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:34.876574 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:34.876581 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:34.876629 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:34.879766 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:34.880737 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:34.880741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:34.880743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:34.880904 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:34.881442 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:34.881445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:34.881447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:34.881768 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:34.881902 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:34.882930 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0927 22:13:34.883195 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":256,"threads_started":1}
I20260927 22:13:34.884864 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:34.863824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35799 (local address 127.0.97.1:47659)
0927 22:13:34.864594 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:34.864644 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:34.866673 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:34.867453 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:34.867462 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:34.868139 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:13:34.869013 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:34.869044 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:34.870441 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:34.870445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:34.871842 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:34.871855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:34.872919 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:34.872923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:34.873130 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:34.873850 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:13:34.873872 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:13:34.876328 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:13:34.879916 (+ 3588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:13:34.879928 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:13:34.879950 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:13:34.880620 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:13:34.881055 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:13:34.881063 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:13:34.881069 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:13:34.881329 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:13:34.881902 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:13:34.881912 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:13:34.882395 (+ 483us) client_negotiation.cc:769] Sending connection context
0927 22:13:34.882743 (+ 348us) client_negotiation.cc:240] Negotiation successful
0927 22:13:34.883359 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":291,"threads_started":1}
I20260927 22:13:34.886564 17734 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35799
I20260927 22:13:34.886933 17734 heartbeater.cc:464] Registering TS with master...
I20260927 22:13:34.887742 17734 heartbeater.cc:511] Master 127.0.97.62:35799 requested a full tablet report, sending...
I20260927 22:13:34.890332 17544 ts_manager.cc:194] Registered new tserver with Master: 55632bda633d41d6932a807ec1007d00 (127.0.97.1:33829)
I20260927 22:13:34.892719 17544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47659
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:13:34.928375 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35799
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:13:35.270779 17740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:35.271121 17740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:35.271312 17740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:35.279000 17740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:35.279176 17740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:35.279364 17740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:13:35.290058 17740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:35.291523 17740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:35.292945 17740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:35.306908 17745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:35.306866 17748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:35.306820 17746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:35.308173 17740 server_base.cc:1034] running on GCE node
I20260927 22:13:35.308836 17740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:35.309803 17740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:35.311028 17740 hybrid_clock.cc:648] HybridClock initialized: now 1790547215310986 us; error 53 us; skew 500 ppm
I20260927 22:13:35.311354 17740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:13:35 dist-test-slave-mp03 krb5kdc[17499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:13:35.314633 17740 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:13:35.316149 17740 webserver.cc:460] Webserver started at http://127.0.97.2:40363/ using document root <none> and password file <none>
I20260927 22:13:35.316860 17740 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:35.317000 17740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:35.317298 17740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:35.320302 17740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "4f8616c01df449f2af9dac45eb57eb4b"
format_stamp: "Formatted at 2026-09-27 22:13:35 on dist-test-slave-mp03"
server_key: "ae20083015f9691f7c176a8b4c366413"
server_key_iv: "5bc4a11c18723dc71febb5d9b43ff25b"
server_key_version: "encryptionkey@0"
I20260927 22:13:35.321110 17740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4f8616c01df449f2af9dac45eb57eb4b"
format_stamp: "Formatted at 2026-09-27 22:13:35 on dist-test-slave-mp03"
server_key: "ae20083015f9691f7c176a8b4c366413"
server_key_iv: "5bc4a11c18723dc71febb5d9b43ff25b"
server_key_version: "encryptionkey@0"
I20260927 22:13:35.326570 17740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:13:35.330190 17755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:35.331668 17740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:13:35.331845 17740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "4f8616c01df449f2af9dac45eb57eb4b"
format_stamp: "Formatted at 2026-09-27 22:13:35 on dist-test-slave-mp03"
server_key: "ae20083015f9691f7c176a8b4c366413"
server_key_iv: "5bc4a11c18723dc71febb5d9b43ff25b"
server_key_version: "encryptionkey@0"
I20260927 22:13:35.332046 17740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:35.347309 17740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:35.351475 17740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:35.351780 17740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:35.352787 17740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:13:35.354244 17740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:13:35.354348 17740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:35.354506 17740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:13:35.354583 17740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:35.389394 17740 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37783
I20260927 22:13:35.389484 17867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37783 every 8 connection(s)
I20260927 22:13:35.390821 17740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:13:35.392385 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 17740
I20260927 22:13:35.392567 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:13:35.393007 388 external_mini_cluster.cc:1428] Setting key 840a221a3fd34335563d40a1661c4e39
Sep 27 22:13:35 dist-test-slave-mp03 krb5kdc[17499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:13:35.413400 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:35.395213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39885 (local address 127.0.97.62:35799)
0927 22:13:35.395659 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:35.395663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:35.396838 (+ 1175us) server_negotiation.cc:407] Connection header received
0927 22:13:35.398757 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:35.398763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:35.398865 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:35.398982 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:35.400557 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:35.401286 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:35.402952 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:35.403378 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:35.403514 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:35.407396 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:35.407401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:35.407404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:35.407441 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:35.410152 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:35.411004 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:35.411016 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:35.411020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:35.411145 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:35.411631 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:35.411643 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:35.411645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:35.411912 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:35.412080 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:35.412785 (+ 705us) server_negotiation.cc:299] Negotiation successful
0927 22:13:35.413014 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":248,"threads_started":1}
I20260927 22:13:35.414419 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:35.395882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35799 (local address 127.0.97.2:39885)
0927 22:13:35.396686 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:35.396730 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:35.398441 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:35.399240 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:35.399249 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:35.399794 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:13:35.400377 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:35.400394 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:35.401460 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:35.401465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:35.402812 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:35.402824 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:35.403619 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:35.403626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:35.403861 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:35.404601 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:13:35.404625 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:13:35.407145 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:13:35.410315 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:13:35.410325 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:13:35.410345 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:13:35.410875 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:13:35.411299 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:13:35.411304 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:13:35.411308 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:13:35.411507 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:13:35.412038 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:13:35.412046 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:13:35.412406 (+ 360us) client_negotiation.cc:769] Sending connection context
0927 22:13:35.412687 (+ 281us) client_negotiation.cc:240] Negotiation successful
0927 22:13:35.413234 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":338,"threads_started":1}
I20260927 22:13:35.416026 17868 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35799
I20260927 22:13:35.416358 17868 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:13:35.417140 17868 heartbeater.cc:511] Master 127.0.97.62:35799 requested a full tablet report, sending...
I20260927 22:13:35.418752 17544 ts_manager.cc:194] Registered new tserver with Master: 4f8616c01df449f2af9dac45eb57eb4b (127.0.97.2:37783)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 22:13:35.419834 17544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:13:35.449415 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35799
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:13:35.786732 17874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:35.787032 17874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:35.787221 17874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:35.794657 17874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:35.794809 17874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:35.794993 17874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:13:35.804989 17874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:35.806443 17874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:35.807794 17874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:35.820873 17880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:35.821028 17879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:35.820996 17882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:35.821813 17874 server_base.cc:1034] running on GCE node
I20260927 22:13:35.822399 17874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:35.823364 17874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:35.824586 17874 hybrid_clock.cc:648] HybridClock initialized: now 1790547215824532 us; error 64 us; skew 500 ppm
I20260927 22:13:35.824904 17874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:13:35 dist-test-slave-mp03 krb5kdc[17499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:13:35.828153 17874 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:13:35.829613 17874 webserver.cc:460] Webserver started at http://127.0.97.3:43551/ using document root <none> and password file <none>
I20260927 22:13:35.830317 17874 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:35.830456 17874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:35.830823 17874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:35.833752 17874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "8e291cfc4da04daa99f70cc68a407731"
format_stamp: "Formatted at 2026-09-27 22:13:35 on dist-test-slave-mp03"
server_key: "918fdb6dfa89415f4fbb2387fb4178b4"
server_key_iv: "bdfb692f7e18627af891a26acd03205d"
server_key_version: "encryptionkey@0"
I20260927 22:13:35.834546 17874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8e291cfc4da04daa99f70cc68a407731"
format_stamp: "Formatted at 2026-09-27 22:13:35 on dist-test-slave-mp03"
server_key: "918fdb6dfa89415f4fbb2387fb4178b4"
server_key_iv: "bdfb692f7e18627af891a26acd03205d"
server_key_version: "encryptionkey@0"
I20260927 22:13:35.839680 17874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:13:35.843536 17889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:35.844924 17874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:13:35.845148 17874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "8e291cfc4da04daa99f70cc68a407731"
format_stamp: "Formatted at 2026-09-27 22:13:35 on dist-test-slave-mp03"
server_key: "918fdb6dfa89415f4fbb2387fb4178b4"
server_key_iv: "bdfb692f7e18627af891a26acd03205d"
server_key_version: "encryptionkey@0"
I20260927 22:13:35.845366 17874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:35.883170 17874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:35.887894 17874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:35.888198 17874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:35.889263 17874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:13:35.890661 17874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:13:35.890769 17874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:35.890909 17874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:13:35.890985 17874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:35.896220 17734 heartbeater.cc:503] Master 127.0.97.62:35799 was elected leader, sending a full tablet report...
I20260927 22:13:35.924858 17874 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36621
I20260927 22:13:35.924968 18001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36621 every 8 connection(s)
I20260927 22:13:35.926153 17874 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:13:35.933740 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 17874
I20260927 22:13:35.933944 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:13:35.934638 388 external_mini_cluster.cc:1428] Setting key bba5f147d0a36b75659109add16b529e
Sep 27 22:13:35 dist-test-slave-mp03 krb5kdc[17499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:13:35.946651 18004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:35.928879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55129 (local address 127.0.97.62:35799)
0927 22:13:35.929332 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:35.929337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:35.930241 (+ 904us) server_negotiation.cc:407] Connection header received
0927 22:13:35.932162 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:35.932166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:35.932240 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:35.932339 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:35.933801 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:35.935078 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:35.937187 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:35.937694 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:35.937828 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:35.941257 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:35.941262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:35.941265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:35.941298 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:35.943921 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:35.944613 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:35.944617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:35.944619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:35.944732 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:35.945095 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:35.945098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:35.945100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:35.945372 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:35.945495 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:35.946126 (+ 631us) server_negotiation.cc:299] Negotiation successful
0927 22:13:35.946308 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":222,"threads_started":1}
I20260927 22:13:35.947682 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:35.929409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35799 (local address 127.0.97.3:55129)
0927 22:13:35.930081 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:35.930122 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:35.931881 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:35.932549 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:35.932557 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:35.933082 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:13:35.933629 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:35.933644 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:35.935289 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:35.935295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:35.937050 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:35.937069 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:35.937925 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:35.937933 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:35.938164 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:35.938908 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:13:35.938929 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:13:35.941076 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:13:35.944054 (+ 2978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:13:35.944061 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:13:35.944081 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:13:35.944484 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:13:35.944826 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:13:35.944829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:13:35.944831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:13:35.945000 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:13:35.945491 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:13:35.945496 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:13:35.945811 (+ 315us) client_negotiation.cc:769] Sending connection context
0927 22:13:35.946041 (+ 230us) client_negotiation.cc:240] Negotiation successful
0927 22:13:35.946428 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":271,"threads_started":1}
I20260927 22:13:35.949239 18002 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35799
I20260927 22:13:35.949553 18002 heartbeater.cc:464] Registering TS with master...
I20260927 22:13:35.950412 18002 heartbeater.cc:511] Master 127.0.97.62:35799 requested a full tablet report, sending...
I20260927 22:13:35.951831 17544 ts_manager.cc:194] Registered new tserver with Master: 8e291cfc4da04daa99f70cc68a407731 (127.0.97.3:36621)
I20260927 22:13:35.952639 17544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55129
I20260927 22:13:35.962522 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:13:35.980646 18004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:35.966883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57502 (local address 127.0.97.62:35799)
0927 22:13:35.967059 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:35.967064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:35.967566 (+ 502us) server_negotiation.cc:407] Connection header received
0927 22:13:35.967673 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:35.967677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:35.967763 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:35.967860 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:35.969390 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:35.970135 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:35.971470 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:35.971893 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:35.972005 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:35.974469 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:35.974474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:35.974478 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:35.974514 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:35.977624 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:35.978643 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:35.978649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:35.978653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:35.978806 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:35.979251 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:35.979255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:35.979257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:35.979598 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:35.979783 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:35.980094 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 22:13:35.980315 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:13:35.984694 17544 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:13:35.991833 17544 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:13:36.027034 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:36.005001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37783 (local address 127.0.0.1:54190)
0927 22:13:36.019210 (+ 14209us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:36.019224 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:36.019367 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:36.020288 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:36.020294 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:36.020319 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:36.020722 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:36.020734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.022467 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.022473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:36.024464 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:36.024485 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.025864 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.025870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:36.026046 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:36.026102 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 22:13:36.026288 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 22:13:36.026493 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14094,"mutex_wait_us":1,"thread_start_us":10508,"threads_started":1}
I20260927 22:13:36.027220 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:36.005594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54190 (local address 127.0.97.2:37783)
0927 22:13:36.013879 (+ 8285us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:36.013889 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:36.019341 (+ 5452us) server_negotiation.cc:407] Connection header received
0927 22:13:36.019554 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:36.019561 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:36.019754 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:36.019977 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:36.020902 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.022326 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.024702 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.025683 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.025837 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:36.026050 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:36.026695 (+ 645us) server_negotiation.cc:299] Negotiation successful
0927 22:13:36.026873 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8126,"thread_start_us":7970,"threads_started":1}
I20260927 22:13:36.027736 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:36.004532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33829 (local address 127.0.0.1:58974)
0927 22:13:36.018549 (+ 14017us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:36.018569 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:36.018703 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:36.019723 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:36.019727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:36.019752 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:36.020163 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:36.020176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.021721 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.021729 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:36.024677 (+ 2948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:36.024688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.026678 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.026687 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:36.026868 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:36.026901 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:13:36.027015 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 22:13:36.027140 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13863,"mutex_wait_us":30,"thread_start_us":14408,"threads_started":1}
I20260927 22:13:36.028093 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:36.005063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58974 (local address 127.0.97.1:33829)
0927 22:13:36.018959 (+ 13896us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:36.018968 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:36.018994 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:13:36.019089 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:36.019098 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:36.019339 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:36.019562 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:36.020367 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.021591 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.024824 (+ 3233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.026220 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.026340 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:36.026553 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:36.027581 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0927 22:13:36.027756 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13709,"thread_start_us":13551,"threads_started":1}
I20260927 22:13:36.028483 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:36.004397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36621 (local address 127.0.0.1:60826)
0927 22:13:36.016000 (+ 11603us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:36.016052 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:36.016323 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:36.018999 (+ 2676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:36.019013 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:36.019063 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:36.019554 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:36.019568 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.022070 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.022075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:36.024447 (+ 2372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:36.024460 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.026300 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.026305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:36.026476 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:36.026505 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 22:13:36.026659 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 22:13:36.026856 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11277,"mutex_wait_us":53,"thread_start_us":13789,"threads_started":1}
I20260927 22:13:36.029084 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:36.004610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60826 (local address 127.0.97.3:36621)
0927 22:13:36.018252 (+ 13642us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:36.018261 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:36.018296 (+ 35us) server_negotiation.cc:407] Connection header received
0927 22:13:36.018401 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:36.018410 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:36.018725 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:36.019594 (+ 869us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:36.020134 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.021928 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.024598 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.026080 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.026228 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:36.026461 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:36.027134 (+ 673us) server_negotiation.cc:299] Negotiation successful
0927 22:13:36.028741 (+ 1607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13423,"thread_start_us":13218,"threads_started":1}
I20260927 22:13:36.031490 17669 tablet_service.cc:1467] Processing CreateTablet for tablet 334afc023c844557a67b49f421554ca5 (DEFAULT_TABLE table=test-table [id=18eb3c21692b4b77b46d65c930b54a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:13:36.031469 17803 tablet_service.cc:1467] Processing CreateTablet for tablet 334afc023c844557a67b49f421554ca5 (DEFAULT_TABLE table=test-table [id=18eb3c21692b4b77b46d65c930b54a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:13:36.032866 17669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 334afc023c844557a67b49f421554ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:36.032904 17803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 334afc023c844557a67b49f421554ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:36.032611 17937 tablet_service.cc:1467] Processing CreateTablet for tablet 334afc023c844557a67b49f421554ca5 (DEFAULT_TABLE table=test-table [id=18eb3c21692b4b77b46d65c930b54a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:13:36.033928 17937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 334afc023c844557a67b49f421554ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:36.044396 18021 tablet_bootstrap.cc:492] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00: Bootstrap starting.
I20260927 22:13:36.047617 18023 tablet_bootstrap.cc:492] T 334afc023c844557a67b49f421554ca5 P 8e291cfc4da04daa99f70cc68a407731: Bootstrap starting.
I20260927 22:13:36.049429 18021 tablet_bootstrap.cc:654] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:36.050755 18021 log.cc:824] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:36.051168 18023 tablet_bootstrap.cc:654] T 334afc023c844557a67b49f421554ca5 P 8e291cfc4da04daa99f70cc68a407731: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:36.052644 18023 log.cc:824] T 334afc023c844557a67b49f421554ca5 P 8e291cfc4da04daa99f70cc68a407731: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:36.054049 18022 tablet_bootstrap.cc:492] T 334afc023c844557a67b49f421554ca5 P 4f8616c01df449f2af9dac45eb57eb4b: Bootstrap starting.
I20260927 22:13:36.057269 18021 tablet_bootstrap.cc:492] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00: No bootstrap required, opened a new log
I20260927 22:13:36.057577 18021 ts_tablet_manager.cc:1397] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s
I20260927 22:13:36.057621 18022 tablet_bootstrap.cc:654] T 334afc023c844557a67b49f421554ca5 P 4f8616c01df449f2af9dac45eb57eb4b: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:36.058873 18022 log.cc:824] T 334afc023c844557a67b49f421554ca5 P 4f8616c01df449f2af9dac45eb57eb4b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:36.059684 18023 tablet_bootstrap.cc:492] T 334afc023c844557a67b49f421554ca5 P 8e291cfc4da04daa99f70cc68a407731: No bootstrap required, opened a new log
I20260927 22:13:36.059991 18023 ts_tablet_manager.cc:1397] T 334afc023c844557a67b49f421554ca5 P 8e291cfc4da04daa99f70cc68a407731: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.001s
I20260927 22:13:36.061934 18022 tablet_bootstrap.cc:492] T 334afc023c844557a67b49f421554ca5 P 4f8616c01df449f2af9dac45eb57eb4b: No bootstrap required, opened a new log
I20260927 22:13:36.062206 18022 ts_tablet_manager.cc:1397] T 334afc023c844557a67b49f421554ca5 P 4f8616c01df449f2af9dac45eb57eb4b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260927 22:13:36.068331 18023 raft_consensus.cc:348] T 334afc023c844557a67b49f421554ca5 P 8e291cfc4da04daa99f70cc68a407731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e291cfc4da04daa99f70cc68a407731" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36621 } } peers { permanent_uuid: "55632bda633d41d6932a807ec1007d00" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33829 } } peers { permanent_uuid: "4f8616c01df449f2af9dac45eb57eb4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37783 } }
I20260927 22:13:36.068722 18023 raft_consensus.cc:374] T 334afc023c844557a67b49f421554ca5 P 8e291cfc4da04daa99f70cc68a407731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:36.068928 18023 raft_consensus.cc:729] T 334afc023c844557a67b49f421554ca5 P 8e291cfc4da04daa99f70cc68a407731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e291cfc4da04daa99f70cc68a407731, State: Initialized, Role: FOLLOWER
I20260927 22:13:36.068820 18021 raft_consensus.cc:348] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e291cfc4da04daa99f70cc68a407731" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36621 } } peers { permanent_uuid: "55632bda633d41d6932a807ec1007d00" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33829 } } peers { permanent_uuid: "4f8616c01df449f2af9dac45eb57eb4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37783 } }
I20260927 22:13:36.069236 18021 raft_consensus.cc:374] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:36.069398 18021 raft_consensus.cc:729] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55632bda633d41d6932a807ec1007d00, State: Initialized, Role: FOLLOWER
I20260927 22:13:36.069629 18023 consensus_queue.cc:260] T 334afc023c844557a67b49f421554ca5 P 8e291cfc4da04daa99f70cc68a407731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e291cfc4da04daa99f70cc68a407731" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36621 } } peers { permanent_uuid: "55632bda633d41d6932a807ec1007d00" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33829 } } peers { permanent_uuid: "4f8616c01df449f2af9dac45eb57eb4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37783 } }
I20260927 22:13:36.069993 18021 consensus_queue.cc:260] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e291cfc4da04daa99f70cc68a407731" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36621 } } peers { permanent_uuid: "55632bda633d41d6932a807ec1007d00" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33829 } } peers { permanent_uuid: "4f8616c01df449f2af9dac45eb57eb4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37783 } }
I20260927 22:13:36.070819 18002 heartbeater.cc:503] Master 127.0.97.62:35799 was elected leader, sending a full tablet report...
I20260927 22:13:36.071498 18023 ts_tablet_manager.cc:1428] T 334afc023c844557a67b49f421554ca5 P 8e291cfc4da04daa99f70cc68a407731: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260927 22:13:36.072014 18021 ts_tablet_manager.cc:1428] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260927 22:13:36.074098 18022 raft_consensus.cc:348] T 334afc023c844557a67b49f421554ca5 P 4f8616c01df449f2af9dac45eb57eb4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e291cfc4da04daa99f70cc68a407731" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36621 } } peers { permanent_uuid: "55632bda633d41d6932a807ec1007d00" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33829 } } peers { permanent_uuid: "4f8616c01df449f2af9dac45eb57eb4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37783 } }
I20260927 22:13:36.074433 18022 raft_consensus.cc:374] T 334afc023c844557a67b49f421554ca5 P 4f8616c01df449f2af9dac45eb57eb4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:36.074548 18022 raft_consensus.cc:729] T 334afc023c844557a67b49f421554ca5 P 4f8616c01df449f2af9dac45eb57eb4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8616c01df449f2af9dac45eb57eb4b, State: Initialized, Role: FOLLOWER
I20260927 22:13:36.075165 18022 consensus_queue.cc:260] T 334afc023c844557a67b49f421554ca5 P 4f8616c01df449f2af9dac45eb57eb4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e291cfc4da04daa99f70cc68a407731" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36621 } } peers { permanent_uuid: "55632bda633d41d6932a807ec1007d00" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33829 } } peers { permanent_uuid: "4f8616c01df449f2af9dac45eb57eb4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37783 } }
I20260927 22:13:36.076319 17868 heartbeater.cc:503] Master 127.0.97.62:35799 was elected leader, sending a full tablet report...
I20260927 22:13:36.077028 18022 ts_tablet_manager.cc:1428] T 334afc023c844557a67b49f421554ca5 P 4f8616c01df449f2af9dac45eb57eb4b: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
W20260927 22:13:36.112978 17735 tablet.cc:2369] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:13:36.140182 18028 raft_consensus.cc:482] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:13:36.140535 18028 raft_consensus.cc:504] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e291cfc4da04daa99f70cc68a407731" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36621 } } peers { permanent_uuid: "55632bda633d41d6932a807ec1007d00" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33829 } } peers { permanent_uuid: "4f8616c01df449f2af9dac45eb57eb4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37783 } }
I20260927 22:13:36.142433 18028 leader_election.cc:288] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e291cfc4da04daa99f70cc68a407731 (127.0.97.3:36621), 4f8616c01df449f2af9dac45eb57eb4b (127.0.97.2:37783)
W20260927 22:13:36.142906 17869 tablet.cc:2369] T 334afc023c844557a67b49f421554ca5 P 4f8616c01df449f2af9dac45eb57eb4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:13:36.151019 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:36.142829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37053 (local address 127.0.97.3:36621)
0927 22:13:36.143034 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:36.143042 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:36.144407 (+ 1365us) server_negotiation.cc:407] Connection header received
0927 22:13:36.144563 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:36.144569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:36.144636 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:36.144725 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:36.145268 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.146300 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.148664 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.149970 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.150083 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:36.150174 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:36.150589 (+ 415us) server_negotiation.cc:299] Negotiation successful
0927 22:13:36.150742 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:13:36.151019 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:36.143293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48593 (local address 127.0.97.2:37783)
0927 22:13:36.143483 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:36.143488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:36.143676 (+ 188us) server_negotiation.cc:407] Connection header received
0927 22:13:36.143917 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:36.143926 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:36.144020 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:36.144107 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:36.144906 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.145985 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.148611 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.149847 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.150023 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:36.150129 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:36.150608 (+ 479us) server_negotiation.cc:299] Negotiation successful
0927 22:13:36.150742 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:13:36.151283 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:36.142734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37783 (local address 127.0.97.1:48593)
0927 22:13:36.143572 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:36.143600 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:36.143743 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:36.144326 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:36.144334 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:36.144366 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:36.144729 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:36.144740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.146125 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.146129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:36.148390 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:36.148409 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.150157 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.150167 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:36.150373 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:36.150446 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 22:13:36.150543 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 22:13:36.150777 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":87,"thread_start_us":750,"threads_started":1}
I20260927 22:13:36.151324 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:36.142811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36621 (local address 127.0.97.1:37053)
0927 22:13:36.144296 (+ 1485us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:36.144312 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:36.144422 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:36.144757 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:36.144760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:36.144790 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:36.145110 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:36.145117 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.146424 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.146430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:36.148510 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:36.148520 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.150205 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.150210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:36.150377 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:36.150447 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 22:13:36.150543 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 22:13:36.150777 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1348,"mutex_wait_us":90,"thread_start_us":1159,"threads_started":1}
I20260927 22:13:36.152290 17823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334afc023c844557a67b49f421554ca5" candidate_uuid: "55632bda633d41d6932a807ec1007d00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8616c01df449f2af9dac45eb57eb4b" is_pre_election: true
I20260927 22:13:36.152290 17957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334afc023c844557a67b49f421554ca5" candidate_uuid: "55632bda633d41d6932a807ec1007d00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e291cfc4da04daa99f70cc68a407731" is_pre_election: true
I20260927 22:13:36.152892 17957 raft_consensus.cc:2454] T 334afc023c844557a67b49f421554ca5 P 8e291cfc4da04daa99f70cc68a407731 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55632bda633d41d6932a807ec1007d00 in term 0.
I20260927 22:13:36.152892 17823 raft_consensus.cc:2454] T 334afc023c844557a67b49f421554ca5 P 4f8616c01df449f2af9dac45eb57eb4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55632bda633d41d6932a807ec1007d00 in term 0.
I20260927 22:13:36.153636 17625 leader_election.cc:302] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8616c01df449f2af9dac45eb57eb4b, 55632bda633d41d6932a807ec1007d00; no voters:
I20260927 22:13:36.154114 18028 raft_consensus.cc:2781] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:13:36.154270 18028 raft_consensus.cc:482] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:13:36.154387 18028 raft_consensus.cc:3037] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:36.156682 18028 raft_consensus.cc:504] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e291cfc4da04daa99f70cc68a407731" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36621 } } peers { permanent_uuid: "55632bda633d41d6932a807ec1007d00" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33829 } } peers { permanent_uuid: "4f8616c01df449f2af9dac45eb57eb4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37783 } }
I20260927 22:13:36.157310 18028 leader_election.cc:288] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00 [CANDIDATE]: Term 1 election: Requested vote from peers 8e291cfc4da04daa99f70cc68a407731 (127.0.97.3:36621), 4f8616c01df449f2af9dac45eb57eb4b (127.0.97.2:37783)
I20260927 22:13:36.157920 17823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334afc023c844557a67b49f421554ca5" candidate_uuid: "55632bda633d41d6932a807ec1007d00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8616c01df449f2af9dac45eb57eb4b"
I20260927 22:13:36.157979 17957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334afc023c844557a67b49f421554ca5" candidate_uuid: "55632bda633d41d6932a807ec1007d00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e291cfc4da04daa99f70cc68a407731"
I20260927 22:13:36.158149 17823 raft_consensus.cc:3037] T 334afc023c844557a67b49f421554ca5 P 4f8616c01df449f2af9dac45eb57eb4b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:36.158207 17957 raft_consensus.cc:3037] T 334afc023c844557a67b49f421554ca5 P 8e291cfc4da04daa99f70cc68a407731 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:36.160482 17823 raft_consensus.cc:2454] T 334afc023c844557a67b49f421554ca5 P 4f8616c01df449f2af9dac45eb57eb4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55632bda633d41d6932a807ec1007d00 in term 1.
I20260927 22:13:36.160679 17957 raft_consensus.cc:2454] T 334afc023c844557a67b49f421554ca5 P 8e291cfc4da04daa99f70cc68a407731 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55632bda633d41d6932a807ec1007d00 in term 1.
I20260927 22:13:36.160986 17625 leader_election.cc:302] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8616c01df449f2af9dac45eb57eb4b, 55632bda633d41d6932a807ec1007d00; no voters:
I20260927 22:13:36.161342 18028 raft_consensus.cc:2781] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:13:36.161784 18028 raft_consensus.cc:686] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00 [term 1 LEADER]: Becoming Leader. State: Replica: 55632bda633d41d6932a807ec1007d00, State: Running, Role: LEADER
I20260927 22:13:36.162322 18028 consensus_queue.cc:237] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e291cfc4da04daa99f70cc68a407731" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36621 } } peers { permanent_uuid: "55632bda633d41d6932a807ec1007d00" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33829 } } peers { permanent_uuid: "4f8616c01df449f2af9dac45eb57eb4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37783 } }
I20260927 22:13:36.167002 17543 catalog_manager.cc:5495] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55632bda633d41d6932a807ec1007d00 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "55632bda633d41d6932a807ec1007d00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e291cfc4da04daa99f70cc68a407731" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55632bda633d41d6932a807ec1007d00" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f8616c01df449f2af9dac45eb57eb4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37783 } health_report { overall_health: UNKNOWN } } }
W20260927 22:13:36.177850 18003 tablet.cc:2369] T 334afc023c844557a67b49f421554ca5 P 8e291cfc4da04daa99f70cc68a407731: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:13:36.221032 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:36.215551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58986 (local address 127.0.97.1:33829)
0927 22:13:36.215757 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:36.215761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:36.215779 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:13:36.216006 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:36.216011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:36.216108 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:36.216236 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:13:36.216628 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.217366 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.218818 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.219229 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.219324 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:36.219857 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:13:36.220396 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:13:36.220548 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:36.220596 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 22:13:36.220673 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:13:36.229696 17823 raft_consensus.cc:1261] T 334afc023c844557a67b49f421554ca5 P 4f8616c01df449f2af9dac45eb57eb4b [term 1 FOLLOWER]: Refusing update from remote peer 55632bda633d41d6932a807ec1007d00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:13:36.229682 17957 raft_consensus.cc:1261] T 334afc023c844557a67b49f421554ca5 P 8e291cfc4da04daa99f70cc68a407731 [term 1 FOLLOWER]: Refusing update from remote peer 55632bda633d41d6932a807ec1007d00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:13:36.230800 18032 consensus_queue.cc:1035] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8616c01df449f2af9dac45eb57eb4b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:13:36.231190 18028 consensus_queue.cc:1035] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e291cfc4da04daa99f70cc68a407731" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:13:36.247679 18035 mvcc.cc:205] Tried to move back new op lower bound from 7334081397666291712 to 7334081397404807168. Current Snapshot: MvccSnapshot[applied={T|T < 7334081397666291712}]
I20260927 22:13:36.262147 17544 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:13:36.262405 17544 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:13:36.265812 17544 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0ab93e0c74c40e691c3608e987f8270: Sending DeleteTablet for 3 replicas of tablet 334afc023c844557a67b49f421554ca5
I20260927 22:13:36.266942 17937 tablet_service.cc:1514] Processing DeleteTablet for tablet 334afc023c844557a67b49f421554ca5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:13:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60826
I20260927 22:13:36.266994 17669 tablet_service.cc:1514] Processing DeleteTablet for tablet 334afc023c844557a67b49f421554ca5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:13:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58974
I20260927 22:13:36.267134 17803 tablet_service.cc:1514] Processing DeleteTablet for tablet 334afc023c844557a67b49f421554ca5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:13:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54190
I20260927 22:13:36.268918 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 17606
I20260927 22:13:36.285161 18046 tablet_replica.cc:330] T 334afc023c844557a67b49f421554ca5 P 8e291cfc4da04daa99f70cc68a407731: stopping tablet replica
I20260927 22:13:36.288661 18046 raft_consensus.cc:2229] T 334afc023c844557a67b49f421554ca5 P 8e291cfc4da04daa99f70cc68a407731 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:13:36.289796 18046 raft_consensus.cc:2258] T 334afc023c844557a67b49f421554ca5 P 8e291cfc4da04daa99f70cc68a407731 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:13:36.296272 18046 ts_tablet_manager.cc:1905] T 334afc023c844557a67b49f421554ca5 P 8e291cfc4da04daa99f70cc68a407731: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:13:36.297859 18045 tablet_replica.cc:330] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00: stopping tablet replica
I20260927 22:13:36.298282 18045 raft_consensus.cc:2229] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:13:36.298950 18045 raft_consensus.cc:2258] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:13:36.301427 18045 ts_tablet_manager.cc:1905] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 22:13:36 dist-test-slave-mp03 krb5kdc[17499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:13:36.303248 18047 tablet_replica.cc:330] T 334afc023c844557a67b49f421554ca5 P 4f8616c01df449f2af9dac45eb57eb4b: stopping tablet replica
I20260927 22:13:36.303670 18047 raft_consensus.cc:2229] T 334afc023c844557a67b49f421554ca5 P 4f8616c01df449f2af9dac45eb57eb4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:13:36.304203 18047 raft_consensus.cc:2258] T 334afc023c844557a67b49f421554ca5 P 4f8616c01df449f2af9dac45eb57eb4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:13:36.309494 18045 ts_tablet_manager.cc:1918] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:13:36.309705 18045 log.cc:1196] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-0/wal/wals/334afc023c844557a67b49f421554ca5
I20260927 22:13:36.310340 18045 ts_tablet_manager.cc:1939] T 334afc023c844557a67b49f421554ca5 P 55632bda633d41d6932a807ec1007d00: Deleting consensus metadata
I20260927 22:13:36.311620 18047 ts_tablet_manager.cc:1905] T 334afc023c844557a67b49f421554ca5 P 4f8616c01df449f2af9dac45eb57eb4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:13:36.312354 18046 ts_tablet_manager.cc:1918] T 334afc023c844557a67b49f421554ca5 P 8e291cfc4da04daa99f70cc68a407731: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:13:36.312441 17529 catalog_manager.cc:4841] TS 55632bda633d41d6932a807ec1007d00 (127.0.97.1:33829): tablet 334afc023c844557a67b49f421554ca5 (table test-table [id=18eb3c21692b4b77b46d65c930b54a36]) successfully deleted
I20260927 22:13:36.312654 18046 log.cc:1196] T 334afc023c844557a67b49f421554ca5 P 8e291cfc4da04daa99f70cc68a407731: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/334afc023c844557a67b49f421554ca5
I20260927 22:13:36.316380 18046 ts_tablet_manager.cc:1939] T 334afc023c844557a67b49f421554ca5 P 8e291cfc4da04daa99f70cc68a407731: Deleting consensus metadata
I20260927 22:13:36.318513 17530 catalog_manager.cc:4841] TS 8e291cfc4da04daa99f70cc68a407731 (127.0.97.3:36621): tablet 334afc023c844557a67b49f421554ca5 (table test-table [id=18eb3c21692b4b77b46d65c930b54a36]) successfully deleted
I20260927 22:13:36.319938 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:36.270227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58990 (local address 127.0.97.1:33829)
0927 22:13:36.270395 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:36.270401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:36.281676 (+ 11275us) server_negotiation.cc:407] Connection header received
0927 22:13:36.282012 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:36.282022 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:36.282148 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:36.282243 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:36.285478 (+ 3235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.287159 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.292451 (+ 5292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.293440 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.293579 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:36.312962 (+ 19383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:36.312972 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:36.312990 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:36.313035 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:36.316108 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:36.317083 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:36.317092 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:36.317096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:36.317289 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:36.317819 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:36.317823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:36.317826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:36.318291 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:36.318570 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:36.319243 (+ 673us) server_negotiation.cc:299] Negotiation successful
0927 22:13:36.319480 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:13:36.320544 18047 ts_tablet_manager.cc:1918] T 334afc023c844557a67b49f421554ca5 P 4f8616c01df449f2af9dac45eb57eb4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:13:36.320729 17729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:36.320781 18047 log.cc:1196] T 334afc023c844557a67b49f421554ca5 P 4f8616c01df449f2af9dac45eb57eb4b: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/334afc023c844557a67b49f421554ca5
I20260927 22:13:36.321285 18047 ts_tablet_manager.cc:1939] T 334afc023c844557a67b49f421554ca5 P 4f8616c01df449f2af9dac45eb57eb4b: Deleting consensus metadata
I20260927 22:13:36.323624 17531 catalog_manager.cc:4841] TS 4f8616c01df449f2af9dac45eb57eb4b (127.0.97.2:37783): tablet 334afc023c844557a67b49f421554ca5 (table test-table [id=18eb3c21692b4b77b46d65c930b54a36]) successfully deleted
I20260927 22:13:36.423053 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 17606
I20260927 22:13:36.441921 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 17740
Sep 27 22:13:36 dist-test-slave-mp03 krb5kdc[17499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:13:36.457798 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:36.442806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54200 (local address 127.0.97.2:37783)
0927 22:13:36.442994 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:36.442998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:36.443019 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:13:36.443082 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:36.443086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:36.443164 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:36.443267 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:36.444598 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.445429 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.446860 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.447283 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.447389 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:36.451784 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:36.451799 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:36.451816 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:36.451856 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:36.455081 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:36.455749 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:36.455756 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:36.455759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:36.455890 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:36.456215 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:36.456219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:36.456221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:36.456629 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:36.456862 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:36.457276 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 22:13:36.457514 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:13:36.458860 17863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:36.550315 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 17740
I20260927 22:13:36.567373 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 17874
Sep 27 22:13:36 dist-test-slave-mp03 krb5kdc[17499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:13:36.583666 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:36.568156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60840 (local address 127.0.97.3:36621)
0927 22:13:36.568342 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:36.568348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:36.568444 (+ 96us) server_negotiation.cc:407] Connection header received
0927 22:13:36.568756 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:36.568761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:36.568828 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:36.568935 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:36.570293 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.571096 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.572387 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:36.572775 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:36.572903 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:36.577910 (+ 5007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:36.577926 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:36.577943 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:36.577983 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:36.580919 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:36.581695 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:36.581701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:36.581703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:36.581839 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:36.582229 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:36.582232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:36.582234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:36.582566 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:36.582863 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:36.583168 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 22:13:36.583328 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:13:36.584285 17997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:36.693522 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 17874
I20260927 22:13:36.709971 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 17512
I20260927 22:13:36.711236 17574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:36.822410 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 17512
2026-09-27T22:13:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4411 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T22:13:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:13:36Z Disabled control of system clock
I20260927 22:13:36.898314 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33341
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:13:37.229290 18056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:37.229595 18056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:37.229678 18056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:37.237010 18056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:13:37.237130 18056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:37.237169 18056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:37.237219 18056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:13:37.237260 18056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:13:37.246488 18056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33341
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:37.247831 18056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:37.249575 18056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:37.261487 18061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:37.261498 18062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:37.261765 18064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:37.262369 18056 server_base.cc:1034] running on GCE node
I20260927 22:13:37.263320 18056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:37.265316 18056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:37.266558 18056 hybrid_clock.cc:648] HybridClock initialized: now 1790547217266496 us; error 50 us; skew 500 ppm
I20260927 22:13:37.266942 18056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:13:37.277124 18056 webserver.cc:460] Webserver started at http://127.0.97.62:38107/ using document root <none> and password file <none>
I20260927 22:13:37.277884 18056 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:37.278020 18056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:37.278373 18056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:37.281518 18056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "4627f8acfc8844ef9788e9d580682123"
format_stamp: "Formatted at 2026-09-27 22:13:37 on dist-test-slave-mp03"
server_key: "49b4b5ecc0ab2022a56a073ece3daf96"
server_key_iv: "2d755f4d414446de0237566544051a39"
server_key_version: "encryptionkey@0"
I20260927 22:13:37.282379 18056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "4627f8acfc8844ef9788e9d580682123"
format_stamp: "Formatted at 2026-09-27 22:13:37 on dist-test-slave-mp03"
server_key: "49b4b5ecc0ab2022a56a073ece3daf96"
server_key_iv: "2d755f4d414446de0237566544051a39"
server_key_version: "encryptionkey@0"
I20260927 22:13:37.287660 18056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:13:37.291384 18070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:37.292871 18056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:13:37.293102 18056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "4627f8acfc8844ef9788e9d580682123"
format_stamp: "Formatted at 2026-09-27 22:13:37 on dist-test-slave-mp03"
server_key: "49b4b5ecc0ab2022a56a073ece3daf96"
server_key_iv: "2d755f4d414446de0237566544051a39"
server_key_version: "encryptionkey@0"
I20260927 22:13:37.293346 18056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:37.308213 18056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:37.309173 18056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:37.309501 18056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:37.329542 18056 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33341
I20260927 22:13:37.329562 18121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33341 every 8 connection(s)
I20260927 22:13:37.331111 18056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:13:37.331919 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 18056
I20260927 22:13:37.332109 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:13:37.332703 388 external_mini_cluster.cc:1428] Setting key 639e9fc6ea810a088f402d14e41785bc
I20260927 22:13:37.336025 18122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:37.346107 18125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:37.334784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55216 (local address 127.0.97.62:33341)
0927 22:13:37.335849 (+ 1065us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:37.335867 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:37.335906 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:13:37.337564 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:37.337585 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:37.338028 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:37.338418 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:37.339016 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:37.340310 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:37.341688 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:37.342350 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:37.342545 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:37.343770 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:37.343777 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:37.343792 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:37.344352 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:37.344429 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:37.344450 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:37.344585 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:37.344827 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 22:13:37.345140 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":409,"threads_started":1}
I20260927 22:13:37.348264 18122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4627f8acfc8844ef9788e9d580682123: Bootstrap starting.
I20260927 22:13:37.351912 18122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4627f8acfc8844ef9788e9d580682123: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:37.352978 18122 log.cc:824] T 00000000000000000000000000000000 P 4627f8acfc8844ef9788e9d580682123: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:37.355973 18122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4627f8acfc8844ef9788e9d580682123: No bootstrap required, opened a new log
I20260927 22:13:37.364048 18122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4627f8acfc8844ef9788e9d580682123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4627f8acfc8844ef9788e9d580682123" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33341 } }
I20260927 22:13:37.364589 18122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4627f8acfc8844ef9788e9d580682123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:37.364713 18122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4627f8acfc8844ef9788e9d580682123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4627f8acfc8844ef9788e9d580682123, State: Initialized, Role: FOLLOWER
I20260927 22:13:37.365321 18122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4627f8acfc8844ef9788e9d580682123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4627f8acfc8844ef9788e9d580682123" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33341 } }
I20260927 22:13:37.365586 18122 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4627f8acfc8844ef9788e9d580682123 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:13:37.365744 18122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4627f8acfc8844ef9788e9d580682123 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:13:37.365934 18122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4627f8acfc8844ef9788e9d580682123 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:37.368376 18122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4627f8acfc8844ef9788e9d580682123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4627f8acfc8844ef9788e9d580682123" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33341 } }
I20260927 22:13:37.368927 18122 leader_election.cc:302] T 00000000000000000000000000000000 P 4627f8acfc8844ef9788e9d580682123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4627f8acfc8844ef9788e9d580682123; no voters:
I20260927 22:13:37.369616 18122 leader_election.cc:288] T 00000000000000000000000000000000 P 4627f8acfc8844ef9788e9d580682123 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:13:37.369745 18127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4627f8acfc8844ef9788e9d580682123 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:13:37.371047 18127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4627f8acfc8844ef9788e9d580682123 [term 1 LEADER]: Becoming Leader. State: Replica: 4627f8acfc8844ef9788e9d580682123, State: Running, Role: LEADER
I20260927 22:13:37.371541 18127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4627f8acfc8844ef9788e9d580682123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4627f8acfc8844ef9788e9d580682123" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33341 } }
I20260927 22:13:37.371909 18122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4627f8acfc8844ef9788e9d580682123 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:13:37.374423 18129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4627f8acfc8844ef9788e9d580682123 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4627f8acfc8844ef9788e9d580682123. Latest consensus state: current_term: 1 leader_uuid: "4627f8acfc8844ef9788e9d580682123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4627f8acfc8844ef9788e9d580682123" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33341 } } }
I20260927 22:13:37.374708 18128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4627f8acfc8844ef9788e9d580682123 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4627f8acfc8844ef9788e9d580682123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4627f8acfc8844ef9788e9d580682123" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33341 } } }
I20260927 22:13:37.374891 18129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4627f8acfc8844ef9788e9d580682123 [sys.catalog]: This master's current role is: LEADER
I20260927 22:13:37.374958 18128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4627f8acfc8844ef9788e9d580682123 [sys.catalog]: This master's current role is: LEADER
I20260927 22:13:37.377015 18134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:13:37.381579 18134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:13:37.390698 18134 catalog_manager.cc:1347] Generated new cluster ID: 45ec4aef0bdd481198c892952046bbfc
I20260927 22:13:37.390825 18134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:13:37.401907 18134 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:13:37.403249 18134 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:13:37.410651 18134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4627f8acfc8844ef9788e9d580682123: Generated new TSK 0
I20260927 22:13:37.411535 18134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:13:37.415557 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33341
--builtin_ntp_servers=127.0.97.20:35759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:13:37.746397 18146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:37.746742 18146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:37.746912 18146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:37.754879 18146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:37.754992 18146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:37.755167 18146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:13:37.764623 18146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:37.765920 18146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:37.767647 18146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:37.780969 18151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:37.781075 18154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:37.781006 18152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:37.781733 18146 server_base.cc:1034] running on GCE node
I20260927 22:13:37.782346 18146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:37.783401 18146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:37.784636 18146 hybrid_clock.cc:648] HybridClock initialized: now 1790547217784576 us; error 53 us; skew 500 ppm
I20260927 22:13:37.784972 18146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:13:37.787384 18146 webserver.cc:460] Webserver started at http://127.0.97.1:40593/ using document root <none> and password file <none>
I20260927 22:13:37.788122 18146 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:37.788235 18146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:37.788568 18146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:37.791550 18146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "0729f71b36c444a1b18dcb802743a929"
format_stamp: "Formatted at 2026-09-27 22:13:37 on dist-test-slave-mp03"
server_key: "a2d742c4bb970f7804505500d77b613a"
server_key_iv: "0e1af2a719715b6076acf8eb033093f2"
server_key_version: "encryptionkey@0"
I20260927 22:13:37.792387 18146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0729f71b36c444a1b18dcb802743a929"
format_stamp: "Formatted at 2026-09-27 22:13:37 on dist-test-slave-mp03"
server_key: "a2d742c4bb970f7804505500d77b613a"
server_key_iv: "0e1af2a719715b6076acf8eb033093f2"
server_key_version: "encryptionkey@0"
I20260927 22:13:37.797749 18146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260927 22:13:37.801476 18160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:37.803063 18146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:13:37.803236 18146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "0729f71b36c444a1b18dcb802743a929"
format_stamp: "Formatted at 2026-09-27 22:13:37 on dist-test-slave-mp03"
server_key: "a2d742c4bb970f7804505500d77b613a"
server_key_iv: "0e1af2a719715b6076acf8eb033093f2"
server_key_version: "encryptionkey@0"
I20260927 22:13:37.803454 18146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:37.823321 18146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:37.824196 18146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:37.824486 18146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:37.825559 18146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:13:37.827174 18146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:13:37.827267 18146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:37.827400 18146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:13:37.827499 18146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:37.860877 18146 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40929
I20260927 22:13:37.860908 18272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40929 every 8 connection(s)
I20260927 22:13:37.862236 18146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:13:37.869563 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 18146
I20260927 22:13:37.869781 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:13:37.870272 388 external_mini_cluster.cc:1428] Setting key 88fd68ee91bd25522e7a7f2afd514b10
I20260927 22:13:37.873615 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33341
--builtin_ntp_servers=127.0.97.20:35759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:13:37.877122 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:37.865062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59647 (local address 127.0.97.62:33341)
0927 22:13:37.865526 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:37.865530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:37.866846 (+ 1316us) server_negotiation.cc:407] Connection header received
0927 22:13:37.868677 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:37.868683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:37.868781 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:37.868904 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:37.870122 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:37.871069 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:37.873466 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:37.873987 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:37.874085 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:37.875935 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:37.875942 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:37.875967 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:37.875991 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:37.876055 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:37.876072 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:37.876175 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:37.876472 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 22:13:37.876601 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":226,"threads_started":1}
I20260927 22:13:37.878013 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:37.865670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33341 (local address 127.0.97.1:59647)
0927 22:13:37.866586 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:37.866735 (+ 149us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:37.868395 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:37.869167 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:37.869176 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:37.869271 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:13:37.869955 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:37.869974 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:37.871255 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:37.871265 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:37.873300 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:37.873320 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:37.874231 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:37.874242 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:37.874656 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:37.875547 (+ 891us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:13:37.875553 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:13:37.875613 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:13:37.875733 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:13:37.876226 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:13:37.876239 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:13:37.876250 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 22:13:37.876419 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 22:13:37.876829 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":276,"threads_started":1}
I20260927 22:13:37.879674 18273 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33341
I20260927 22:13:37.879995 18273 heartbeater.cc:464] Registering TS with master...
I20260927 22:13:37.880759 18273 heartbeater.cc:511] Master 127.0.97.62:33341 requested a full tablet report, sending...
I20260927 22:13:37.883010 18087 ts_manager.cc:194] Registered new tserver with Master: 0729f71b36c444a1b18dcb802743a929 (127.0.97.1:40929)
I20260927 22:13:37.885367 18087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:59647
W20260927 22:13:38.215272 18277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:38.215587 18277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:38.215680 18277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:38.226480 18277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:38.226649 18277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:38.226843 18277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:13:38.236368 18277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:38.237648 18277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:38.239521 18277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:38.252921 18282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:38.252889 18285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:38.252977 18283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:38.253870 18277 server_base.cc:1034] running on GCE node
I20260927 22:13:38.254503 18277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:38.255462 18277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:38.256654 18277 hybrid_clock.cc:648] HybridClock initialized: now 1790547218256608 us; error 44 us; skew 500 ppm
I20260927 22:13:38.256935 18277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:13:38.259320 18277 webserver.cc:460] Webserver started at http://127.0.97.2:35733/ using document root <none> and password file <none>
I20260927 22:13:38.259990 18277 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:38.260075 18277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:38.260372 18277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:38.263259 18277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "201a682a6297407890784f18ef9a358c"
format_stamp: "Formatted at 2026-09-27 22:13:38 on dist-test-slave-mp03"
server_key: "b09beebacfd776571e67b899aa3e5e22"
server_key_iv: "2652c49928481a0cd93cec1ac47ff9ba"
server_key_version: "encryptionkey@0"
I20260927 22:13:38.264020 18277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "201a682a6297407890784f18ef9a358c"
format_stamp: "Formatted at 2026-09-27 22:13:38 on dist-test-slave-mp03"
server_key: "b09beebacfd776571e67b899aa3e5e22"
server_key_iv: "2652c49928481a0cd93cec1ac47ff9ba"
server_key_version: "encryptionkey@0"
I20260927 22:13:38.269081 18277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 22:13:38.272735 18291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:38.274279 18277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:13:38.274456 18277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "201a682a6297407890784f18ef9a358c"
format_stamp: "Formatted at 2026-09-27 22:13:38 on dist-test-slave-mp03"
server_key: "b09beebacfd776571e67b899aa3e5e22"
server_key_iv: "2652c49928481a0cd93cec1ac47ff9ba"
server_key_version: "encryptionkey@0"
I20260927 22:13:38.274803 18277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:38.293351 18277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:38.294219 18277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:38.294502 18277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:38.295756 18277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:13:38.297369 18277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:13:38.297461 18277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:38.297696 18277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:13:38.297792 18277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:38.335120 18277 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42483
I20260927 22:13:38.335165 18403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42483 every 8 connection(s)
I20260927 22:13:38.336648 18277 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:13:38.346062 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 18277
I20260927 22:13:38.346274 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:13:38.346853 388 external_mini_cluster.cc:1428] Setting key 9ab1c490e5fd5c7d344d92b380147408
I20260927 22:13:38.352998 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:38.339532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50095 (local address 127.0.97.62:33341)
0927 22:13:38.339756 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:38.339763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:38.341456 (+ 1693us) server_negotiation.cc:407] Connection header received
0927 22:13:38.344181 (+ 2725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:38.344186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:38.344254 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:38.344350 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:38.345901 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:38.347147 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:38.349648 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:38.350345 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:38.350484 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:38.352061 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:38.352067 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:38.352072 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:38.352094 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:38.352134 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:38.352150 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:38.352258 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:38.352537 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 22:13:38.352636 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:13:38.354107 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:38.340135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33341 (local address 127.0.97.2:50095)
0927 22:13:38.341260 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:38.341317 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:38.343841 (+ 2524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:38.344624 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:38.344636 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:38.344756 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:13:38.345676 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:38.345702 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:38.347317 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:38.347323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:38.349482 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:38.349497 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:38.350578 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:38.350583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:38.350868 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:38.351667 (+ 799us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:13:38.351670 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:13:38.351709 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:13:38.351829 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:13:38.352306 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:13:38.352316 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:13:38.352323 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:13:38.352497 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 22:13:38.352969 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":340,"threads_started":1}
I20260927 22:13:38.354095 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33341
--builtin_ntp_servers=127.0.97.20:35759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:13:38.355562 18404 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33341
I20260927 22:13:38.355899 18404 heartbeater.cc:464] Registering TS with master...
I20260927 22:13:38.356679 18404 heartbeater.cc:511] Master 127.0.97.62:33341 requested a full tablet report, sending...
I20260927 22:13:38.358150 18087 ts_manager.cc:194] Registered new tserver with Master: 201a682a6297407890784f18ef9a358c (127.0.97.2:42483)
I20260927 22:13:38.359133 18087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:50095
W20260927 22:13:38.690135 18407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:38.690416 18407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:38.690500 18407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:38.697872 18407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:38.698006 18407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:38.698143 18407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:13:38.707454 18407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:38.708894 18407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:38.710772 18407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:38.724484 18415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:38.724565 18412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:38.724592 18413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:38.725502 18407 server_base.cc:1034] running on GCE node
I20260927 22:13:38.726219 18407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:38.727267 18407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:38.728468 18407 hybrid_clock.cc:648] HybridClock initialized: now 1790547218728422 us; error 37 us; skew 500 ppm
I20260927 22:13:38.728802 18407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:13:38.731240 18407 webserver.cc:460] Webserver started at http://127.0.97.3:34841/ using document root <none> and password file <none>
I20260927 22:13:38.731946 18407 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:38.732077 18407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:38.732403 18407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:38.735299 18407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "7e330bffd1aa46ac8bf9d631964d9702"
format_stamp: "Formatted at 2026-09-27 22:13:38 on dist-test-slave-mp03"
server_key: "02004732154ee402dbf0bf2b2876ec99"
server_key_iv: "7e27a6b3853d580d49d377cd081d70f6"
server_key_version: "encryptionkey@0"
I20260927 22:13:38.736093 18407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7e330bffd1aa46ac8bf9d631964d9702"
format_stamp: "Formatted at 2026-09-27 22:13:38 on dist-test-slave-mp03"
server_key: "02004732154ee402dbf0bf2b2876ec99"
server_key_iv: "7e27a6b3853d580d49d377cd081d70f6"
server_key_version: "encryptionkey@0"
I20260927 22:13:38.741266 18407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 22:13:38.744642 18421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:38.746052 18407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:13:38.746273 18407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "7e330bffd1aa46ac8bf9d631964d9702"
format_stamp: "Formatted at 2026-09-27 22:13:38 on dist-test-slave-mp03"
server_key: "02004732154ee402dbf0bf2b2876ec99"
server_key_iv: "7e27a6b3853d580d49d377cd081d70f6"
server_key_version: "encryptionkey@0"
I20260927 22:13:38.746541 18407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:38.765941 18407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:38.767009 18407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:38.767338 18407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:38.768399 18407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:13:38.770073 18407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:13:38.770211 18407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:38.770372 18407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:13:38.770471 18407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:38.806380 18407 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42505
I20260927 22:13:38.806399 18533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42505 every 8 connection(s)
I20260927 22:13:38.807902 18407 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:13:38.810529 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 18407
I20260927 22:13:38.810772 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:13:38.811260 388 external_mini_cluster.cc:1428] Setting key 282a6d183f64ce28f1da9501025cc6b3
I20260927 22:13:38.821043 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:38.810510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52303 (local address 127.0.97.62:33341)
0927 22:13:38.810763 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:38.810770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:38.812442 (+ 1672us) server_negotiation.cc:407] Connection header received
0927 22:13:38.814790 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:38.814795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:38.814882 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:38.814996 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:38.816072 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:38.816814 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:38.818377 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:38.818823 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:38.818945 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:38.820206 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:38.820212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:38.820216 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:38.820247 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:38.820275 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:38.820286 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:38.820377 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:38.820582 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 22:13:38.820682 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 22:13:38.822050 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:38.811140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33341 (local address 127.0.97.3:52303)
0927 22:13:38.812280 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:38.812333 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:38.814483 (+ 2150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:38.815243 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:38.815252 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:38.815346 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:13:38.815913 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:38.815928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:38.816965 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:38.816971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:38.818240 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:38.818251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:38.819041 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:38.819048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:38.819283 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:38.819913 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:13:38.819915 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:13:38.819953 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:13:38.820031 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:13:38.820374 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:13:38.820386 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:13:38.820396 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 22:13:38.820574 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 22:13:38.820982 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":384,"threads_started":1}
I20260927 22:13:38.823422 18534 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33341
I20260927 22:13:38.823783 18534 heartbeater.cc:464] Registering TS with master...
I20260927 22:13:38.824646 18534 heartbeater.cc:511] Master 127.0.97.62:33341 requested a full tablet report, sending...
I20260927 22:13:38.826231 18087 ts_manager.cc:194] Registered new tserver with Master: 7e330bffd1aa46ac8bf9d631964d9702 (127.0.97.3:42505)
I20260927 22:13:38.827183 18087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:52303
I20260927 22:13:38.839181 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:13:38.849677 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:38.843224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53610 (local address 127.0.97.62:33341)
0927 22:13:38.843521 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:38.843530 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:38.843832 (+ 302us) server_negotiation.cc:407] Connection header received
0927 22:13:38.844065 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:38.844072 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:38.844168 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:38.844304 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:38.844884 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:38.845663 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:38.847298 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:38.847707 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:38.847846 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:38.848958 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:38.848963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:38.848966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:38.848985 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:38.849017 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:38.849029 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:38.849136 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:38.849290 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 22:13:38.849385 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260927 22:13:38.853762 18087 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:13:38.860828 18087 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:13:38.888520 18273 heartbeater.cc:503] Master 127.0.97.62:33341 was elected leader, sending a full tablet report...
I20260927 22:13:38.895460 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:38.873397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40929 (local address 127.0.0.1:38580)
0927 22:13:38.882671 (+ 9274us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:38.882715 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:38.882930 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:38.886302 (+ 3372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:38.886313 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:38.886355 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:38.886881 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:38.886899 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:38.889722 (+ 2823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:38.889727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:38.892085 (+ 2358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:38.892099 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:38.894372 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:38.894384 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:38.894585 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:38.894686 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 22:13:38.894803 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 22:13:38.895050 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8933,"mutex_wait_us":47,"thread_start_us":8785,"threads_started":1}
I20260927 22:13:38.895507 18550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:38.874004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38580 (local address 127.0.97.1:40929)
0927 22:13:38.885395 (+ 11391us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:38.885407 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:38.885449 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:13:38.885582 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:38.885593 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:38.885956 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:38.886232 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:38.887137 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:38.889550 (+ 2413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:38.892295 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:38.893833 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:38.893963 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:38.894232 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:38.894980 (+ 748us) server_negotiation.cc:299] Negotiation successful
0927 22:13:38.895150 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11115,"thread_start_us":11189,"threads_started":1}
I20260927 22:13:38.896015 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:38.874664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49308 (local address 127.0.97.3:42505)
0927 22:13:38.887767 (+ 13103us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:38.887775 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:38.887803 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:13:38.887874 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:38.887880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:38.888065 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:38.888265 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:38.888913 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:38.890662 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:38.892952 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:38.893967 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:38.894109 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:38.894378 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:38.895466 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0927 22:13:38.895648 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12934,"thread_start_us":12797,"threads_started":1}
I20260927 22:13:38.896369 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:38.873901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42505 (local address 127.0.0.1:49308)
0927 22:13:38.887009 (+ 13108us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:38.887026 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:38.887156 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:38.888251 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:38.888255 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:38.888280 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:38.888688 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:38.888702 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:38.890813 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:38.890818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:38.892758 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:38.892769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:38.894500 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:38.894506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:38.894724 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:38.894761 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:13:38.894846 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 22:13:38.895059 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12969,"mutex_wait_us":1,"thread_start_us":12816,"threads_started":1}
I20260927 22:13:38.896932 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:38.873426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42483 (local address 127.0.0.1:43196)
0927 22:13:38.884923 (+ 11497us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:38.884947 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:38.885097 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:38.889964 (+ 4867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:38.889969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:38.889993 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:38.890354 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:38.890364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:38.892257 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:38.892262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:38.894317 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:38.894332 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:38.896202 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:38.896208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:38.896398 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:38.896436 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 22:13:38.896522 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 22:13:38.896645 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11329,"thread_start_us":11150,"threads_started":1}
I20260927 22:13:38.897454 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:38.873965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43196 (local address 127.0.97.2:42483)
0927 22:13:38.888907 (+ 14942us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:38.888916 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:38.888943 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:13:38.889040 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:38.889063 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:38.889301 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:38.889743 (+ 442us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:38.890529 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:38.892088 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:38.894538 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:38.895997 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:38.896171 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:38.896401 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:38.896943 (+ 542us) server_negotiation.cc:299] Negotiation successful
0927 22:13:38.897133 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14770,"thread_start_us":14791,"threads_started":1}
I20260927 22:13:38.900271 18208 tablet_service.cc:1467] Processing CreateTablet for tablet eca2a5e649c54e618ab8fe51968ab379 (DEFAULT_TABLE table=test-table [id=58819037545b43a5acdf59c55a202a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:13:38.900789 18339 tablet_service.cc:1467] Processing CreateTablet for tablet eca2a5e649c54e618ab8fe51968ab379 (DEFAULT_TABLE table=test-table [id=58819037545b43a5acdf59c55a202a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:13:38.901906 18208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca2a5e649c54e618ab8fe51968ab379. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:38.901412 18469 tablet_service.cc:1467] Processing CreateTablet for tablet eca2a5e649c54e618ab8fe51968ab379 (DEFAULT_TABLE table=test-table [id=58819037545b43a5acdf59c55a202a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:13:38.902060 18339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca2a5e649c54e618ab8fe51968ab379. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:38.902776 18469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca2a5e649c54e618ab8fe51968ab379. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:38.914398 18552 tablet_bootstrap.cc:492] T eca2a5e649c54e618ab8fe51968ab379 P 201a682a6297407890784f18ef9a358c: Bootstrap starting.
I20260927 22:13:38.917061 18553 tablet_bootstrap.cc:492] T eca2a5e649c54e618ab8fe51968ab379 P 0729f71b36c444a1b18dcb802743a929: Bootstrap starting.
I20260927 22:13:38.918114 18552 tablet_bootstrap.cc:654] T eca2a5e649c54e618ab8fe51968ab379 P 201a682a6297407890784f18ef9a358c: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:38.918802 18554 tablet_bootstrap.cc:492] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702: Bootstrap starting.
I20260927 22:13:38.920657 18552 log.cc:824] T eca2a5e649c54e618ab8fe51968ab379 P 201a682a6297407890784f18ef9a358c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:38.920858 18553 tablet_bootstrap.cc:654] T eca2a5e649c54e618ab8fe51968ab379 P 0729f71b36c444a1b18dcb802743a929: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:38.921918 18553 log.cc:824] T eca2a5e649c54e618ab8fe51968ab379 P 0729f71b36c444a1b18dcb802743a929: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:38.922094 18554 tablet_bootstrap.cc:654] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:38.923293 18554 log.cc:824] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:38.923863 18552 tablet_bootstrap.cc:492] T eca2a5e649c54e618ab8fe51968ab379 P 201a682a6297407890784f18ef9a358c: No bootstrap required, opened a new log
I20260927 22:13:38.924190 18552 ts_tablet_manager.cc:1397] T eca2a5e649c54e618ab8fe51968ab379 P 201a682a6297407890784f18ef9a358c: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 22:13:38.924959 18553 tablet_bootstrap.cc:492] T eca2a5e649c54e618ab8fe51968ab379 P 0729f71b36c444a1b18dcb802743a929: No bootstrap required, opened a new log
I20260927 22:13:38.925218 18553 ts_tablet_manager.cc:1397] T eca2a5e649c54e618ab8fe51968ab379 P 0729f71b36c444a1b18dcb802743a929: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 22:13:38.925765 18554 tablet_bootstrap.cc:492] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702: No bootstrap required, opened a new log
I20260927 22:13:38.925997 18554 ts_tablet_manager.cc:1397] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 22:13:38.933391 18554 raft_consensus.cc:348] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0729f71b36c444a1b18dcb802743a929" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40929 } } peers { permanent_uuid: "201a682a6297407890784f18ef9a358c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42483 } } peers { permanent_uuid: "7e330bffd1aa46ac8bf9d631964d9702" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42505 } }
I20260927 22:13:38.933796 18554 raft_consensus.cc:374] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:38.933912 18554 raft_consensus.cc:729] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e330bffd1aa46ac8bf9d631964d9702, State: Initialized, Role: FOLLOWER
I20260927 22:13:38.934535 18554 consensus_queue.cc:260] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0729f71b36c444a1b18dcb802743a929" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40929 } } peers { permanent_uuid: "201a682a6297407890784f18ef9a358c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42483 } } peers { permanent_uuid: "7e330bffd1aa46ac8bf9d631964d9702" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42505 } }
I20260927 22:13:38.935726 18534 heartbeater.cc:503] Master 127.0.97.62:33341 was elected leader, sending a full tablet report...
I20260927 22:13:38.935531 18553 raft_consensus.cc:348] T eca2a5e649c54e618ab8fe51968ab379 P 0729f71b36c444a1b18dcb802743a929 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0729f71b36c444a1b18dcb802743a929" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40929 } } peers { permanent_uuid: "201a682a6297407890784f18ef9a358c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42483 } } peers { permanent_uuid: "7e330bffd1aa46ac8bf9d631964d9702" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42505 } }
I20260927 22:13:38.935514 18552 raft_consensus.cc:348] T eca2a5e649c54e618ab8fe51968ab379 P 201a682a6297407890784f18ef9a358c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0729f71b36c444a1b18dcb802743a929" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40929 } } peers { permanent_uuid: "201a682a6297407890784f18ef9a358c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42483 } } peers { permanent_uuid: "7e330bffd1aa46ac8bf9d631964d9702" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42505 } }
I20260927 22:13:38.935992 18552 raft_consensus.cc:374] T eca2a5e649c54e618ab8fe51968ab379 P 201a682a6297407890784f18ef9a358c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:38.935992 18553 raft_consensus.cc:374] T eca2a5e649c54e618ab8fe51968ab379 P 0729f71b36c444a1b18dcb802743a929 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:38.936084 18552 raft_consensus.cc:729] T eca2a5e649c54e618ab8fe51968ab379 P 201a682a6297407890784f18ef9a358c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201a682a6297407890784f18ef9a358c, State: Initialized, Role: FOLLOWER
I20260927 22:13:38.936094 18553 raft_consensus.cc:729] T eca2a5e649c54e618ab8fe51968ab379 P 0729f71b36c444a1b18dcb802743a929 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0729f71b36c444a1b18dcb802743a929, State: Initialized, Role: FOLLOWER
I20260927 22:13:38.936436 18554 ts_tablet_manager.cc:1428] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260927 22:13:38.936746 18553 consensus_queue.cc:260] T eca2a5e649c54e618ab8fe51968ab379 P 0729f71b36c444a1b18dcb802743a929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0729f71b36c444a1b18dcb802743a929" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40929 } } peers { permanent_uuid: "201a682a6297407890784f18ef9a358c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42483 } } peers { permanent_uuid: "7e330bffd1aa46ac8bf9d631964d9702" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42505 } }
I20260927 22:13:38.936832 18552 consensus_queue.cc:260] T eca2a5e649c54e618ab8fe51968ab379 P 201a682a6297407890784f18ef9a358c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0729f71b36c444a1b18dcb802743a929" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40929 } } peers { permanent_uuid: "201a682a6297407890784f18ef9a358c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42483 } } peers { permanent_uuid: "7e330bffd1aa46ac8bf9d631964d9702" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42505 } }
I20260927 22:13:38.938336 18404 heartbeater.cc:503] Master 127.0.97.62:33341 was elected leader, sending a full tablet report...
I20260927 22:13:38.938531 18553 ts_tablet_manager.cc:1428] T eca2a5e649c54e618ab8fe51968ab379 P 0729f71b36c444a1b18dcb802743a929: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260927 22:13:38.939893 18552 ts_tablet_manager.cc:1428] T eca2a5e649c54e618ab8fe51968ab379 P 201a682a6297407890784f18ef9a358c: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
W20260927 22:13:39.059815 18535 tablet.cc:2369] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:13:39.088912 18405 tablet.cc:2369] T eca2a5e649c54e618ab8fe51968ab379 P 201a682a6297407890784f18ef9a358c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:13:39.114995 18274 tablet.cc:2369] T eca2a5e649c54e618ab8fe51968ab379 P 0729f71b36c444a1b18dcb802743a929: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:13:39.125666 18558 raft_consensus.cc:482] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:13:39.125953 18558 raft_consensus.cc:504] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0729f71b36c444a1b18dcb802743a929" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40929 } } peers { permanent_uuid: "201a682a6297407890784f18ef9a358c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42483 } } peers { permanent_uuid: "7e330bffd1aa46ac8bf9d631964d9702" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42505 } }
I20260927 22:13:39.127692 18558 leader_election.cc:288] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0729f71b36c444a1b18dcb802743a929 (127.0.97.1:40929), 201a682a6297407890784f18ef9a358c (127.0.97.2:42483)
I20260927 22:13:39.135186 18550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:39.128729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38933 (local address 127.0.97.1:40929)
0927 22:13:39.128931 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:39.128936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:39.128954 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:13:39.129052 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:39.129058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:39.129154 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:39.129252 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:39.129732 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:39.130721 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:39.133160 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:39.134357 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:39.134486 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:39.134624 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:39.134839 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 22:13:39.134976 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:13:39.135284 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:39.127945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40929 (local address 127.0.97.3:38933)
0927 22:13:39.128400 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:39.128421 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:39.128542 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:39.129322 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:39.129326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:39.129351 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:39.129583 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:39.129594 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:39.130837 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:39.130841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:39.132999 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:39.133013 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:39.134506 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:39.134511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:39.134683 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:39.134708 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 22:13:39.134810 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 22:13:39.134953 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327}
I20260927 22:13:39.135586 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:39.128110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42483 (local address 127.0.97.3:35877)
0927 22:13:39.129097 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:39.129111 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:39.129259 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:39.129753 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:39.129758 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:39.129784 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:39.130023 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:39.130029 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:39.131273 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:39.131278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:39.133295 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:39.133304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:39.134873 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:39.134879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:39.135038 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:39.135078 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 22:13:39.135179 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 22:13:39.135289 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"thread_start_us":1155,"threads_started":1}
I20260927 22:13:39.135603 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:39.128291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35877 (local address 127.0.97.2:42483)
0927 22:13:39.128480 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:39.128486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:39.129203 (+ 717us) server_negotiation.cc:407] Connection header received
0927 22:13:39.129425 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:39.129430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:39.129515 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:39.129578 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:39.130184 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:39.131130 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:39.133467 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:39.134713 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:39.134849 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:39.134930 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:39.135214 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 22:13:39.135357 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:13:39.136287 18228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca2a5e649c54e618ab8fe51968ab379" candidate_uuid: "7e330bffd1aa46ac8bf9d631964d9702" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0729f71b36c444a1b18dcb802743a929" is_pre_election: true
I20260927 22:13:39.136652 18359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca2a5e649c54e618ab8fe51968ab379" candidate_uuid: "7e330bffd1aa46ac8bf9d631964d9702" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201a682a6297407890784f18ef9a358c" is_pre_election: true
I20260927 22:13:39.136787 18228 raft_consensus.cc:2454] T eca2a5e649c54e618ab8fe51968ab379 P 0729f71b36c444a1b18dcb802743a929 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e330bffd1aa46ac8bf9d631964d9702 in term 0.
I20260927 22:13:39.137108 18359 raft_consensus.cc:2454] T eca2a5e649c54e618ab8fe51968ab379 P 201a682a6297407890784f18ef9a358c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e330bffd1aa46ac8bf9d631964d9702 in term 0.
I20260927 22:13:39.137459 18423 leader_election.cc:302] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0729f71b36c444a1b18dcb802743a929, 7e330bffd1aa46ac8bf9d631964d9702; no voters:
I20260927 22:13:39.137872 18558 raft_consensus.cc:2781] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:13:39.138029 18558 raft_consensus.cc:482] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:13:39.138136 18558 raft_consensus.cc:3037] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:39.140453 18558 raft_consensus.cc:504] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0729f71b36c444a1b18dcb802743a929" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40929 } } peers { permanent_uuid: "201a682a6297407890784f18ef9a358c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42483 } } peers { permanent_uuid: "7e330bffd1aa46ac8bf9d631964d9702" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42505 } }
I20260927 22:13:39.141057 18558 leader_election.cc:288] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702 [CANDIDATE]: Term 1 election: Requested vote from peers 0729f71b36c444a1b18dcb802743a929 (127.0.97.1:40929), 201a682a6297407890784f18ef9a358c (127.0.97.2:42483)
I20260927 22:13:39.141516 18228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca2a5e649c54e618ab8fe51968ab379" candidate_uuid: "7e330bffd1aa46ac8bf9d631964d9702" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0729f71b36c444a1b18dcb802743a929"
I20260927 22:13:39.141516 18359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca2a5e649c54e618ab8fe51968ab379" candidate_uuid: "7e330bffd1aa46ac8bf9d631964d9702" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201a682a6297407890784f18ef9a358c"
I20260927 22:13:39.141727 18359 raft_consensus.cc:3037] T eca2a5e649c54e618ab8fe51968ab379 P 201a682a6297407890784f18ef9a358c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:39.141727 18228 raft_consensus.cc:3037] T eca2a5e649c54e618ab8fe51968ab379 P 0729f71b36c444a1b18dcb802743a929 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:39.145004 18359 raft_consensus.cc:2454] T eca2a5e649c54e618ab8fe51968ab379 P 201a682a6297407890784f18ef9a358c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e330bffd1aa46ac8bf9d631964d9702 in term 1.
I20260927 22:13:39.145002 18228 raft_consensus.cc:2454] T eca2a5e649c54e618ab8fe51968ab379 P 0729f71b36c444a1b18dcb802743a929 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e330bffd1aa46ac8bf9d631964d9702 in term 1.
I20260927 22:13:39.145416 18423 leader_election.cc:302] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0729f71b36c444a1b18dcb802743a929, 7e330bffd1aa46ac8bf9d631964d9702; no voters:
I20260927 22:13:39.145762 18558 raft_consensus.cc:2781] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:13:39.146245 18558 raft_consensus.cc:686] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702 [term 1 LEADER]: Becoming Leader. State: Replica: 7e330bffd1aa46ac8bf9d631964d9702, State: Running, Role: LEADER
I20260927 22:13:39.146795 18558 consensus_queue.cc:237] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0729f71b36c444a1b18dcb802743a929" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40929 } } peers { permanent_uuid: "201a682a6297407890784f18ef9a358c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42483 } } peers { permanent_uuid: "7e330bffd1aa46ac8bf9d631964d9702" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42505 } }
I20260927 22:13:39.151456 18085 catalog_manager.cc:5495] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e330bffd1aa46ac8bf9d631964d9702 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "7e330bffd1aa46ac8bf9d631964d9702" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0729f71b36c444a1b18dcb802743a929" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "201a682a6297407890784f18ef9a358c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e330bffd1aa46ac8bf9d631964d9702" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42505 } health_report { overall_health: HEALTHY } } }
I20260927 22:13:39.198777 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:39.193170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49324 (local address 127.0.97.3:42505)
0927 22:13:39.193402 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:39.193407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:39.193427 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:13:39.193739 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:39.193744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:39.193831 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:39.193985 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:13:39.194348 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:39.195129 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:39.196547 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:39.196972 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:39.197123 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:39.197650 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:13:39.198188 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:13:39.198355 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:39.198406 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 22:13:39.198486 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 22:13:39.206748 18359 raft_consensus.cc:1261] T eca2a5e649c54e618ab8fe51968ab379 P 201a682a6297407890784f18ef9a358c [term 1 FOLLOWER]: Refusing update from remote peer 7e330bffd1aa46ac8bf9d631964d9702: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:13:39.206766 18228 raft_consensus.cc:1261] T eca2a5e649c54e618ab8fe51968ab379 P 0729f71b36c444a1b18dcb802743a929 [term 1 FOLLOWER]: Refusing update from remote peer 7e330bffd1aa46ac8bf9d631964d9702: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:13:39.207779 18558 consensus_queue.cc:1035] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201a682a6297407890784f18ef9a358c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:13:39.208153 18562 consensus_queue.cc:1035] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0729f71b36c444a1b18dcb802743a929" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:13:39.223024 18565 mvcc.cc:205] Tried to move back new op lower bound from 7334081409860702208 to 7334081409629384704. Current Snapshot: MvccSnapshot[applied={T|T < 7334081409860702208}]
I20260927 22:13:39.234980 18087 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:13:39.235280 18087 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:13:39.238754 18087 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4627f8acfc8844ef9788e9d580682123: Sending DeleteTablet for 3 replicas of tablet eca2a5e649c54e618ab8fe51968ab379
I20260927 22:13:39.239918 18208 tablet_service.cc:1514] Processing DeleteTablet for tablet eca2a5e649c54e618ab8fe51968ab379 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:13:39 UTC) from {username='slave'} at 127.0.0.1:38580
I20260927 22:13:39.239919 18339 tablet_service.cc:1514] Processing DeleteTablet for tablet eca2a5e649c54e618ab8fe51968ab379 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:13:39 UTC) from {username='slave'} at 127.0.0.1:43196
I20260927 22:13:39.240389 18469 tablet_service.cc:1514] Processing DeleteTablet for tablet eca2a5e649c54e618ab8fe51968ab379 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:13:39 UTC) from {username='slave'} at 127.0.0.1:49308
I20260927 22:13:39.244071 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 18146
I20260927 22:13:39.252302 18577 tablet_replica.cc:330] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702: stopping tablet replica
I20260927 22:13:39.252293 18575 tablet_replica.cc:330] T eca2a5e649c54e618ab8fe51968ab379 P 0729f71b36c444a1b18dcb802743a929: stopping tablet replica
I20260927 22:13:39.252910 18575 raft_consensus.cc:2229] T eca2a5e649c54e618ab8fe51968ab379 P 0729f71b36c444a1b18dcb802743a929 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:13:39.253059 18577 raft_consensus.cc:2229] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:13:39.253702 18575 raft_consensus.cc:2258] T eca2a5e649c54e618ab8fe51968ab379 P 0729f71b36c444a1b18dcb802743a929 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:13:39.253949 18577 raft_consensus.cc:2258] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:13:39.256136 18577 ts_tablet_manager.cc:1905] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:13:39.256292 18575 ts_tablet_manager.cc:1905] T eca2a5e649c54e618ab8fe51968ab379 P 0729f71b36c444a1b18dcb802743a929: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:13:39.254575 18576 tablet_replica.cc:330] T eca2a5e649c54e618ab8fe51968ab379 P 201a682a6297407890784f18ef9a358c: stopping tablet replica
I20260927 22:13:39.260939 18576 raft_consensus.cc:2229] T eca2a5e649c54e618ab8fe51968ab379 P 201a682a6297407890784f18ef9a358c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:13:39.261639 18576 raft_consensus.cc:2258] T eca2a5e649c54e618ab8fe51968ab379 P 201a682a6297407890784f18ef9a358c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:13:39.263303 18577 ts_tablet_manager.cc:1918] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:13:39.263514 18577 log.cc:1196] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/eca2a5e649c54e618ab8fe51968ab379
I20260927 22:13:39.264210 18577 ts_tablet_manager.cc:1939] T eca2a5e649c54e618ab8fe51968ab379 P 7e330bffd1aa46ac8bf9d631964d9702: Deleting consensus metadata
I20260927 22:13:39.265332 18576 ts_tablet_manager.cc:1905] T eca2a5e649c54e618ab8fe51968ab379 P 201a682a6297407890784f18ef9a358c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:13:39.266137 18074 catalog_manager.cc:4841] TS 7e330bffd1aa46ac8bf9d631964d9702 (127.0.97.3:42505): tablet eca2a5e649c54e618ab8fe51968ab379 (table test-table [id=58819037545b43a5acdf59c55a202a69]) successfully deleted
I20260927 22:13:39.268486 18575 ts_tablet_manager.cc:1918] T eca2a5e649c54e618ab8fe51968ab379 P 0729f71b36c444a1b18dcb802743a929: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:13:39.268731 18575 log.cc:1196] T eca2a5e649c54e618ab8fe51968ab379 P 0729f71b36c444a1b18dcb802743a929: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-0/wal/wals/eca2a5e649c54e618ab8fe51968ab379
I20260927 22:13:39.269353 18575 ts_tablet_manager.cc:1939] T eca2a5e649c54e618ab8fe51968ab379 P 0729f71b36c444a1b18dcb802743a929: Deleting consensus metadata
I20260927 22:13:39.270143 18550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:39.245831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38592 (local address 127.0.97.1:40929)
0927 22:13:39.246077 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:39.246086 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:39.261376 (+ 15290us) server_negotiation.cc:407] Connection header received
0927 22:13:39.261592 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:39.261598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:39.261707 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:39.261844 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:39.262306 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:39.263514 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:39.266025 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:39.266686 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:39.266819 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:39.268281 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:39.268292 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:39.268309 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:39.269158 (+ 849us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:39.269245 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:39.269279 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:39.269376 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:39.269650 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 22:13:39.269767 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:13:39.271521 18268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:39.271522 18072 catalog_manager.cc:4841] TS 0729f71b36c444a1b18dcb802743a929 (127.0.97.1:40929): tablet eca2a5e649c54e618ab8fe51968ab379 (table test-table [id=58819037545b43a5acdf59c55a202a69]) successfully deleted
I20260927 22:13:39.272801 18576 ts_tablet_manager.cc:1918] T eca2a5e649c54e618ab8fe51968ab379 P 201a682a6297407890784f18ef9a358c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:13:39.273084 18576 log.cc:1196] T eca2a5e649c54e618ab8fe51968ab379 P 201a682a6297407890784f18ef9a358c: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/eca2a5e649c54e618ab8fe51968ab379
I20260927 22:13:39.273746 18576 ts_tablet_manager.cc:1939] T eca2a5e649c54e618ab8fe51968ab379 P 201a682a6297407890784f18ef9a358c: Deleting consensus metadata
I20260927 22:13:39.275621 18073 catalog_manager.cc:4841] TS 201a682a6297407890784f18ef9a358c (127.0.97.2:42483): tablet eca2a5e649c54e618ab8fe51968ab379 (table test-table [id=58819037545b43a5acdf59c55a202a69]) successfully deleted
I20260927 22:13:39.373747 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 18146
I20260927 22:13:39.392140 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 18277
I20260927 22:13:39.400170 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:39.393043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43204 (local address 127.0.97.2:42483)
0927 22:13:39.393280 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:39.393287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:39.393309 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:13:39.393502 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:39.393509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:39.393635 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:39.393751 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:39.394091 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:39.395267 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:39.396749 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:39.397427 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:39.397565 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:39.398814 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:39.398824 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:39.398840 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:39.399425 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:39.399473 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:39.399493 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:39.399585 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:39.399733 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 22:13:39.399816 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 22:13:39.400955 18399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:39.493019 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 18277
I20260927 22:13:39.510538 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 18407
I20260927 22:13:39.518402 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:39.511367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49340 (local address 127.0.97.3:42505)
0927 22:13:39.511547 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:39.511551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:39.511609 (+ 58us) server_negotiation.cc:407] Connection header received
0927 22:13:39.511824 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:39.511827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:39.511934 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:39.512021 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:39.512486 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:39.513279 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:39.515330 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:39.515762 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:39.515857 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:39.517028 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:39.517036 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:39.517050 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:39.517621 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:39.517680 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:39.517698 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:39.517781 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:39.517947 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 22:13:39.518025 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:13:39.519174 18529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:39.639621 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 18407
I20260927 22:13:39.658504 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 18056
I20260927 22:13:39.659626 18117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:39.768078 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 18056
2026-09-27T22:13:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2944 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:13:39 dist-test-slave-mp03 krb5kdc[18585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:13:39 dist-test-slave-mp03 krb5kdc[18585](info): set up 2 sockets
Sep 27 22:13:39 dist-test-slave-mp03 krb5kdc[18585](info): commencing operation
krb5kdc: starting...
W20260927 22:13:40.882833 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:13:41 dist-test-slave-mp03 krb5kdc[18585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:13:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:13:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:13:41.142328 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44847
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44847
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:13:41.477056 18598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:41.477437 18598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:41.477607 18598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:41.485044 18598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:13:41.485198 18598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:41.485307 18598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:41.485414 18598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:13:41.485500 18598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:13:41.495427 18598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:41.496798 18598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:41.498126 18598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:41.510247 18606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:41.510342 18604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:41.510367 18603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:41.511121 18598 server_base.cc:1034] running on GCE node
I20260927 22:13:41.511932 18598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:41.513329 18598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:41.514565 18598 hybrid_clock.cc:648] HybridClock initialized: now 1790547221514516 us; error 59 us; skew 500 ppm
I20260927 22:13:41.514967 18598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:13:41 dist-test-slave-mp03 krb5kdc[18585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:13:41.518019 18598 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:13:41.519542 18598 webserver.cc:460] Webserver started at http://127.0.97.62:34713/ using document root <none> and password file <none>
I20260927 22:13:41.520226 18598 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:41.520370 18598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:41.520715 18598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:41.523721 18598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "f0169298caea4ac7a24c8adb416269ab"
format_stamp: "Formatted at 2026-09-27 22:13:41 on dist-test-slave-mp03"
server_key: "13cb485e42e9c01ed1fb7ae7850576d0"
server_key_iv: "90779833d87c6f96466c8505d8aa6132"
server_key_version: "encryptionkey@0"
I20260927 22:13:41.524477 18598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "f0169298caea4ac7a24c8adb416269ab"
format_stamp: "Formatted at 2026-09-27 22:13:41 on dist-test-slave-mp03"
server_key: "13cb485e42e9c01ed1fb7ae7850576d0"
server_key_iv: "90779833d87c6f96466c8505d8aa6132"
server_key_version: "encryptionkey@0"
I20260927 22:13:41.529649 18598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:13:41.533543 18613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:41.535140 18598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:13:41.535338 18598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "f0169298caea4ac7a24c8adb416269ab"
format_stamp: "Formatted at 2026-09-27 22:13:41 on dist-test-slave-mp03"
server_key: "13cb485e42e9c01ed1fb7ae7850576d0"
server_key_iv: "90779833d87c6f96466c8505d8aa6132"
server_key_version: "encryptionkey@0"
I20260927 22:13:41.535550 18598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:41.551721 18598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:41.555766 18598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:41.556054 18598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:41.574913 18598 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44847
I20260927 22:13:41.574934 18664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44847 every 8 connection(s)
I20260927 22:13:41.576372 18598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:13:41.580335 18665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:41.586210 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 18598
I20260927 22:13:41.586418 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:13:41.586956 388 external_mini_cluster.cc:1428] Setting key 39e1627468c3ea34fbd150cdaf2f5cfa
I20260927 22:13:41.591751 18665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0169298caea4ac7a24c8adb416269ab: Bootstrap starting.
I20260927 22:13:41.595630 18665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0169298caea4ac7a24c8adb416269ab: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:41.596926 18665 log.cc:824] T 00000000000000000000000000000000 P f0169298caea4ac7a24c8adb416269ab: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:41.600244 18665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0169298caea4ac7a24c8adb416269ab: No bootstrap required, opened a new log
Sep 27 22:13:41 dist-test-slave-mp03 krb5kdc[18585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:13:41.608796 18665 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0169298caea4ac7a24c8adb416269ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0169298caea4ac7a24c8adb416269ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44847 } }
I20260927 22:13:41.609207 18665 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0169298caea4ac7a24c8adb416269ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:41.609332 18665 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0169298caea4ac7a24c8adb416269ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0169298caea4ac7a24c8adb416269ab, State: Initialized, Role: FOLLOWER
I20260927 22:13:41.610083 18665 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0169298caea4ac7a24c8adb416269ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0169298caea4ac7a24c8adb416269ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44847 } }
I20260927 22:13:41.610375 18665 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0169298caea4ac7a24c8adb416269ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:13:41.610515 18665 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0169298caea4ac7a24c8adb416269ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:13:41.610764 18665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0169298caea4ac7a24c8adb416269ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:41.611766 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:41.589076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56018 (local address 127.0.97.62:44847)
0927 22:13:41.589946 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:41.589958 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:41.589999 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:13:41.591792 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:41.591817 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:41.592272 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:41.592615 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:41.594590 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:41.596039 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:41.597928 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:41.598777 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:41.598904 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:41.603657 (+ 4753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:41.603665 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:41.603683 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:41.603723 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:41.607049 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:41.607768 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:41.607781 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:41.607788 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:41.607959 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:41.608370 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:41.608375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:41.608378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:41.608936 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:41.609266 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:41.609791 (+ 525us) server_negotiation.cc:299] Negotiation successful
0927 22:13:41.610252 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":426,"threads_started":1}
I20260927 22:13:41.613514 18665 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0169298caea4ac7a24c8adb416269ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0169298caea4ac7a24c8adb416269ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44847 } }
I20260927 22:13:41.614032 18665 leader_election.cc:302] T 00000000000000000000000000000000 P f0169298caea4ac7a24c8adb416269ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0169298caea4ac7a24c8adb416269ab; no voters:
I20260927 22:13:41.614903 18665 leader_election.cc:288] T 00000000000000000000000000000000 P f0169298caea4ac7a24c8adb416269ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:13:41.615056 18670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0169298caea4ac7a24c8adb416269ab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:13:41.616034 18670 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0169298caea4ac7a24c8adb416269ab [term 1 LEADER]: Becoming Leader. State: Replica: f0169298caea4ac7a24c8adb416269ab, State: Running, Role: LEADER
I20260927 22:13:41.616755 18670 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0169298caea4ac7a24c8adb416269ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0169298caea4ac7a24c8adb416269ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44847 } }
I20260927 22:13:41.617292 18665 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0169298caea4ac7a24c8adb416269ab [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:13:41.620276 18672 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0169298caea4ac7a24c8adb416269ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0169298caea4ac7a24c8adb416269ab. Latest consensus state: current_term: 1 leader_uuid: "f0169298caea4ac7a24c8adb416269ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0169298caea4ac7a24c8adb416269ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44847 } } }
I20260927 22:13:41.620705 18672 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0169298caea4ac7a24c8adb416269ab [sys.catalog]: This master's current role is: LEADER
I20260927 22:13:41.620256 18671 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0169298caea4ac7a24c8adb416269ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0169298caea4ac7a24c8adb416269ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0169298caea4ac7a24c8adb416269ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44847 } } }
I20260927 22:13:41.621330 18671 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0169298caea4ac7a24c8adb416269ab [sys.catalog]: This master's current role is: LEADER
I20260927 22:13:41.623262 18678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:13:41.628984 18678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:13:41.638403 18678 catalog_manager.cc:1347] Generated new cluster ID: 6029c86679ca410ca11f5128fe4626de
I20260927 22:13:41.638556 18678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:13:41.661717 18678 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:13:41.663700 18678 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:13:41.670522 18678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0169298caea4ac7a24c8adb416269ab: Generated new TSK 0
I20260927 22:13:41.671589 18678 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:13:41.731681 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44847
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:13:42.067806 18691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:42.068131 18691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:42.068228 18691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:42.076022 18691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:42.076144 18691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:42.076267 18691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:13:42.086493 18691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:42.087934 18691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:42.089237 18691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:42.103052 18697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:42.103156 18696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:42.103142 18699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:42.104419 18691 server_base.cc:1034] running on GCE node
I20260927 22:13:42.105156 18691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:42.106148 18691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:42.107352 18691 hybrid_clock.cc:648] HybridClock initialized: now 1790547222107305 us; error 33 us; skew 500 ppm
I20260927 22:13:42.107798 18691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:13:42 dist-test-slave-mp03 krb5kdc[18585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:13:42.111517 18691 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:13:42.113101 18691 webserver.cc:460] Webserver started at http://127.0.97.1:38681/ using document root <none> and password file <none>
I20260927 22:13:42.113830 18691 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:42.113947 18691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:42.114243 18691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:42.117676 18691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "4ec112f0f39b48a1952d472511622335"
format_stamp: "Formatted at 2026-09-27 22:13:42 on dist-test-slave-mp03"
server_key: "d645d76c4117f95dd3bcefb5fe2188f9"
server_key_iv: "d2ad792f2c04bcd4792461ffabb719fc"
server_key_version: "encryptionkey@0"
I20260927 22:13:42.118719 18691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4ec112f0f39b48a1952d472511622335"
format_stamp: "Formatted at 2026-09-27 22:13:42 on dist-test-slave-mp03"
server_key: "d645d76c4117f95dd3bcefb5fe2188f9"
server_key_iv: "d2ad792f2c04bcd4792461ffabb719fc"
server_key_version: "encryptionkey@0"
I20260927 22:13:42.124691 18691 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 22:13:42.128662 18706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:42.130162 18691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:13:42.130358 18691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "4ec112f0f39b48a1952d472511622335"
format_stamp: "Formatted at 2026-09-27 22:13:42 on dist-test-slave-mp03"
server_key: "d645d76c4117f95dd3bcefb5fe2188f9"
server_key_iv: "d2ad792f2c04bcd4792461ffabb719fc"
server_key_version: "encryptionkey@0"
I20260927 22:13:42.130589 18691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:42.146162 18691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:42.151000 18691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:42.151328 18691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:42.152369 18691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:13:42.153785 18691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:13:42.153889 18691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:42.154030 18691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:13:42.154110 18691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:42.188432 18691 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39765
I20260927 22:13:42.188480 18818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39765 every 8 connection(s)
I20260927 22:13:42.189881 18691 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:13:42.196252 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 18691
I20260927 22:13:42.196435 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:13:42.196990 388 external_mini_cluster.cc:1428] Setting key fc6ffd466b3dd377f996c59fd40ba2d3
Sep 27 22:13:42 dist-test-slave-mp03 krb5kdc[18585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:13:42.211189 18821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:42.192692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55337 (local address 127.0.97.62:44847)
0927 22:13:42.193133 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:42.193137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:42.194237 (+ 1100us) server_negotiation.cc:407] Connection header received
0927 22:13:42.196484 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:42.196490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:42.196587 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:42.196711 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:42.198435 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:42.199240 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:42.200857 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:42.201403 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:42.201540 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:42.205284 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:42.205290 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:42.205293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:42.205325 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:42.208027 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:42.208783 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:42.208795 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:42.208798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:42.208961 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:42.209417 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:42.209422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:42.209426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:42.209697 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:42.209854 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:42.210554 (+ 700us) server_negotiation.cc:299] Negotiation successful
0927 22:13:42.210813 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":206,"threads_started":1}
I20260927 22:13:42.212183 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:42.193272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44847 (local address 127.0.97.1:55337)
0927 22:13:42.194056 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:42.194106 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:42.196146 (+ 2040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:42.196940 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:42.196948 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:42.197655 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:13:42.198252 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:42.198269 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:42.199389 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:42.199396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:42.200710 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:42.200722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:42.201642 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:42.201651 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:42.201890 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:42.202703 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:13:42.202729 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:13:42.205048 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:13:42.208189 (+ 3141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:13:42.208198 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:13:42.208215 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:13:42.208652 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:13:42.209081 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:13:42.209086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:13:42.209090 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:13:42.209297 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:13:42.209828 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:13:42.209836 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:13:42.210180 (+ 344us) client_negotiation.cc:769] Sending connection context
0927 22:13:42.210453 (+ 273us) client_negotiation.cc:240] Negotiation successful
0927 22:13:42.211024 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":293,"threads_started":1}
I20260927 22:13:42.213876 18819 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44847
I20260927 22:13:42.214217 18819 heartbeater.cc:464] Registering TS with master...
I20260927 22:13:42.215135 18819 heartbeater.cc:511] Master 127.0.97.62:44847 requested a full tablet report, sending...
I20260927 22:13:42.217859 18630 ts_manager.cc:194] Registered new tserver with Master: 4ec112f0f39b48a1952d472511622335 (127.0.97.1:39765)
I20260927 22:13:42.220117 18630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55337
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:13:42.251776 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44847
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:13:42.601960 18825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:42.602368 18825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:42.602492 18825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:42.610041 18825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:42.610163 18825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:42.610289 18825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:13:42.620812 18825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:42.622051 18825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:42.623337 18825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:42.637038 18831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:42.637080 18833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:42.637126 18830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:42.638012 18825 server_base.cc:1034] running on GCE node
I20260927 22:13:42.638806 18825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:42.639962 18825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:42.641185 18825 hybrid_clock.cc:648] HybridClock initialized: now 1790547222641131 us; error 48 us; skew 500 ppm
I20260927 22:13:42.641517 18825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:13:42 dist-test-slave-mp03 krb5kdc[18585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:13:42.654515 18825 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:13:42.656173 18825 webserver.cc:460] Webserver started at http://127.0.97.2:40175/ using document root <none> and password file <none>
I20260927 22:13:42.656929 18825 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:42.657075 18825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:42.657420 18825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:42.660405 18825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "89f790ec7c04411a87709fa1b3384036"
format_stamp: "Formatted at 2026-09-27 22:13:42 on dist-test-slave-mp03"
server_key: "b1ecc32498ee0732e69230f0d47b68cf"
server_key_iv: "13c9b0a9d4a3240117558ee75b2439d4"
server_key_version: "encryptionkey@0"
I20260927 22:13:42.661180 18825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "89f790ec7c04411a87709fa1b3384036"
format_stamp: "Formatted at 2026-09-27 22:13:42 on dist-test-slave-mp03"
server_key: "b1ecc32498ee0732e69230f0d47b68cf"
server_key_iv: "13c9b0a9d4a3240117558ee75b2439d4"
server_key_version: "encryptionkey@0"
I20260927 22:13:42.666258 18825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:13:42.669812 18840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:42.671178 18825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:13:42.671379 18825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "89f790ec7c04411a87709fa1b3384036"
format_stamp: "Formatted at 2026-09-27 22:13:42 on dist-test-slave-mp03"
server_key: "b1ecc32498ee0732e69230f0d47b68cf"
server_key_iv: "13c9b0a9d4a3240117558ee75b2439d4"
server_key_version: "encryptionkey@0"
I20260927 22:13:42.671595 18825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:42.691601 18825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:42.696141 18825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:42.696457 18825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:42.697453 18825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:13:42.698848 18825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:13:42.698951 18825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:42.699103 18825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:13:42.699177 18825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:42.732540 18825 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34027
I20260927 22:13:42.732627 18952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34027 every 8 connection(s)
I20260927 22:13:42.733927 18825 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:13:42.737207 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 18825
I20260927 22:13:42.737391 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:13:42.737970 388 external_mini_cluster.cc:1428] Setting key 9bc6e90eb2c42d18ccb81adafe5142e5
Sep 27 22:13:42 dist-test-slave-mp03 krb5kdc[18585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:13:42.754642 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:42.736634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57051 (local address 127.0.97.62:44847)
0927 22:13:42.737117 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:42.737121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:42.738241 (+ 1120us) server_negotiation.cc:407] Connection header received
0927 22:13:42.740389 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:42.740394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:42.740477 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:42.740605 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:42.742092 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:42.742919 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:42.744457 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:42.744904 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:42.745037 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:42.748764 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:42.748778 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:42.748786 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:42.748820 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:42.751502 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:42.752241 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:42.752248 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:42.752252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:42.752390 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:42.752826 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:42.752833 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:42.752836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:42.753111 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:42.753277 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:42.753979 (+ 702us) server_negotiation.cc:299] Negotiation successful
0927 22:13:42.754212 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":258,"threads_started":1}
I20260927 22:13:42.755621 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:42.737215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44847 (local address 127.0.97.2:57051)
0927 22:13:42.738061 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:42.738112 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:42.740128 (+ 2016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:42.740840 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:42.740848 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:42.741350 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:13:42.741925 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:42.741944 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:42.743067 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:42.743077 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:42.744305 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:42.744317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:42.745126 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:42.745133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:42.745358 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:42.746048 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:13:42.746096 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:13:42.748539 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:13:42.751650 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:13:42.751660 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:13:42.751678 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:13:42.752118 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:13:42.752511 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:13:42.752516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:13:42.752520 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:13:42.752705 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:13:42.753251 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:13:42.753259 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:13:42.753618 (+ 359us) client_negotiation.cc:769] Sending connection context
0927 22:13:42.753901 (+ 283us) client_negotiation.cc:240] Negotiation successful
0927 22:13:42.754445 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":315,"threads_started":1}
I20260927 22:13:42.757205 18953 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44847
I20260927 22:13:42.757548 18953 heartbeater.cc:464] Registering TS with master...
I20260927 22:13:42.758563 18953 heartbeater.cc:511] Master 127.0.97.62:44847 requested a full tablet report, sending...
I20260927 22:13:42.760300 18630 ts_manager.cc:194] Registered new tserver with Master: 89f790ec7c04411a87709fa1b3384036 (127.0.97.2:34027)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:13:42.761247 18630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:13:42.793774 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44847
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:13:43.126328 18959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:43.126677 18959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:43.126842 18959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:43.134073 18959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:43.134172 18959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:43.134280 18959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:13:43.144120 18959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:43.145402 18959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:43.146699 18959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:43.159710 18964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:43.159809 18967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:43.159844 18965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:43.160856 18959 server_base.cc:1034] running on GCE node
I20260927 22:13:43.161453 18959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:43.162400 18959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:43.163599 18959 hybrid_clock.cc:648] HybridClock initialized: now 1790547223163533 us; error 63 us; skew 500 ppm
I20260927 22:13:43.163928 18959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:13:43 dist-test-slave-mp03 krb5kdc[18585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:13:43.167053 18959 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:13:43.168505 18959 webserver.cc:460] Webserver started at http://127.0.97.3:39199/ using document root <none> and password file <none>
I20260927 22:13:43.169162 18959 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:43.169256 18959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:43.169520 18959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:43.172410 18959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "c13485ebe7474ff8b73098316ef381b1"
format_stamp: "Formatted at 2026-09-27 22:13:43 on dist-test-slave-mp03"
server_key: "585a2a5b07c68e3ba7c5aeaedc92d637"
server_key_iv: "9dc719caf1205077679808c5653258d2"
server_key_version: "encryptionkey@0"
I20260927 22:13:43.173146 18959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c13485ebe7474ff8b73098316ef381b1"
format_stamp: "Formatted at 2026-09-27 22:13:43 on dist-test-slave-mp03"
server_key: "585a2a5b07c68e3ba7c5aeaedc92d637"
server_key_iv: "9dc719caf1205077679808c5653258d2"
server_key_version: "encryptionkey@0"
I20260927 22:13:43.178241 18959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 22:13:43.181632 18974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:43.183094 18959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:13:43.183264 18959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "c13485ebe7474ff8b73098316ef381b1"
format_stamp: "Formatted at 2026-09-27 22:13:43 on dist-test-slave-mp03"
server_key: "585a2a5b07c68e3ba7c5aeaedc92d637"
server_key_iv: "9dc719caf1205077679808c5653258d2"
server_key_version: "encryptionkey@0"
I20260927 22:13:43.183454 18959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:43.202035 18959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:43.206449 18959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:43.206770 18959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:43.207810 18959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:13:43.209189 18959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:13:43.209260 18959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:43.209444 18959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:13:43.209496 18959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:43.223651 18819 heartbeater.cc:503] Master 127.0.97.62:44847 was elected leader, sending a full tablet report...
I20260927 22:13:43.244380 18959 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44349
I20260927 22:13:43.244417 19086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44349 every 8 connection(s)
I20260927 22:13:43.245664 18959 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:13:43.248531 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 18959
I20260927 22:13:43.248710 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:13:43.249135 388 external_mini_cluster.cc:1428] Setting key 727000712deca4118def8484f6b8fc1d
Sep 27 22:13:43 dist-test-slave-mp03 krb5kdc[18585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:13:43.265654 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:43.248391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49853 (local address 127.0.97.62:44847)
0927 22:13:43.248607 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:43.248613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:43.249888 (+ 1275us) server_negotiation.cc:407] Connection header received
0927 22:13:43.251739 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:43.251744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:43.251836 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:43.251940 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:43.253306 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.254001 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.255680 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.256085 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.256179 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:43.259630 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:43.259635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:43.259638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:43.259670 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:43.262478 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:43.263520 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:43.263524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:43.263527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:43.263646 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:43.264163 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:43.264168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:43.264171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:43.264486 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:43.264605 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:43.265249 (+ 644us) server_negotiation.cc:299] Negotiation successful
0927 22:13:43.265426 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:13:43.266700 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:43.248907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44847 (local address 127.0.97.3:49853)
0927 22:13:43.249731 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:43.249776 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:43.251515 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:43.252116 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:43.252123 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:43.252622 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:13:43.253157 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:43.253174 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.254220 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.254224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:43.255550 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:43.255562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.256263 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.256267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:43.256521 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:43.257281 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:13:43.257303 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:13:43.259439 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:13:43.262632 (+ 3193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:13:43.262641 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:13:43.262658 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:13:43.263376 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:13:43.263762 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:13:43.263768 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:13:43.263771 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:13:43.264027 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:13:43.264602 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:13:43.264608 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:13:43.264927 (+ 319us) client_negotiation.cc:769] Sending connection context
0927 22:13:43.265155 (+ 228us) client_negotiation.cc:240] Negotiation successful
0927 22:13:43.265535 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":358,"threads_started":1}
I20260927 22:13:43.268123 19087 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44847
I20260927 22:13:43.268450 19087 heartbeater.cc:464] Registering TS with master...
I20260927 22:13:43.269207 19087 heartbeater.cc:511] Master 127.0.97.62:44847 requested a full tablet report, sending...
I20260927 22:13:43.270720 18630 ts_manager.cc:194] Registered new tserver with Master: c13485ebe7474ff8b73098316ef381b1 (127.0.97.3:44349)
I20260927 22:13:43.271545 18630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49853
I20260927 22:13:43.276521 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:13:43.294906 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:43.280876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56032 (local address 127.0.97.62:44847)
0927 22:13:43.281141 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:43.281149 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:43.281405 (+ 256us) server_negotiation.cc:407] Connection header received
0927 22:13:43.281640 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:43.281646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:43.281732 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:43.281846 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:43.283531 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.284291 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.285831 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.286257 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.286443 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:43.289443 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:43.289457 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:43.289461 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:43.289493 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:43.292068 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:43.292837 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:43.292844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:43.292853 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:43.292977 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:43.293424 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:43.293432 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:43.293435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:43.293741 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:43.294025 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:43.294392 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 22:13:43.294576 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 22:13:43.299811 18630 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:13:43.306804 18630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:13:43.331372 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:43.319338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44349 (local address 127.0.0.1:46584)
0927 22:13:43.322147 (+ 2809us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:43.322197 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:43.322537 (+ 340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:43.323757 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:43.323762 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:43.323788 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:43.324116 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:43.324124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.326139 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.326147 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:43.328430 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:43.328443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.330232 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.330240 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:43.330424 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:43.330475 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 22:13:43.330761 (+ 286us) client_negotiation.cc:240] Negotiation successful
0927 22:13:43.330968 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2561,"thread_start_us":2370,"threads_started":1}
I20260927 22:13:43.332336 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:43.320061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46584 (local address 127.0.97.3:44349)
0927 22:13:43.320975 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:43.320988 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:43.322316 (+ 1328us) server_negotiation.cc:407] Connection header received
0927 22:13:43.322828 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:43.322837 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:43.323156 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:43.323338 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:43.324314 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.325997 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.328598 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.330029 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.330156 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:43.330380 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:43.331805 (+ 1425us) server_negotiation.cc:299] Negotiation successful
0927 22:13:43.331977 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":544,"threads_started":1}
I20260927 22:13:43.332683 19101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:43.319857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34027 (local address 127.0.0.1:46478)
0927 22:13:43.324687 (+ 4830us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:43.324707 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:43.324910 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:43.325643 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:43.325650 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:43.325682 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:43.326080 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:43.326093 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.328050 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.328059 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:43.329809 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:43.329822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.331873 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.331880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:43.332042 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:43.332080 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 22:13:43.332193 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 22:13:43.332314 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4714,"mutex_wait_us":41,"thread_start_us":4578,"threads_started":1}
I20260927 22:13:43.332768 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:43.320433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46478 (local address 127.0.97.2:34027)
0927 22:13:43.321524 (+ 1091us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:43.321535 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:43.324807 (+ 3272us) server_negotiation.cc:407] Connection header received
0927 22:13:43.325089 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:43.325098 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:43.325342 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:43.325495 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:43.326289 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.327896 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.329964 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.331416 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.331548 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:43.331747 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:43.332255 (+ 508us) server_negotiation.cc:299] Negotiation successful
0927 22:13:43.332430 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":722,"threads_started":1}
I20260927 22:13:43.334033 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:43.319842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53638 (local address 127.0.97.1:39765)
0927 22:13:43.325146 (+ 5304us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:43.325159 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:43.325751 (+ 592us) server_negotiation.cc:407] Connection header received
0927 22:13:43.325907 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:43.325916 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:43.326147 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:43.326336 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:43.327364 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.328701 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.331188 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.332879 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.333152 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:43.333385 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:43.333565 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 22:13:43.333722 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5088,"thread_start_us":4952,"threads_started":1}
I20260927 22:13:43.334007 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:43.319338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39765 (local address 127.0.0.1:53638)
0927 22:13:43.325620 (+ 6282us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:43.325636 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:43.325751 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:43.326641 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:43.326645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:43.326671 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:43.327130 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:43.327143 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.328839 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.328845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:43.331006 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:43.331018 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.333195 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.333201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:43.333373 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:43.333405 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:13:43.333516 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 22:13:43.333631 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6150,"mutex_wait_us":721,"thread_start_us":5302,"threads_started":1}
I20260927 22:13:43.336865 19022 tablet_service.cc:1467] Processing CreateTablet for tablet dbc1c7400e7d448c90e898d00b448839 (DEFAULT_TABLE table=test-table [id=855518b8cc4746509018446dba02618d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:13:43.336864 18888 tablet_service.cc:1467] Processing CreateTablet for tablet dbc1c7400e7d448c90e898d00b448839 (DEFAULT_TABLE table=test-table [id=855518b8cc4746509018446dba02618d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:13:43.337576 18754 tablet_service.cc:1467] Processing CreateTablet for tablet dbc1c7400e7d448c90e898d00b448839 (DEFAULT_TABLE table=test-table [id=855518b8cc4746509018446dba02618d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:13:43.338363 19022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc1c7400e7d448c90e898d00b448839. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:43.338363 18888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc1c7400e7d448c90e898d00b448839. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:43.338887 18754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc1c7400e7d448c90e898d00b448839. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:43.350890 19105 tablet_bootstrap.cc:492] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335: Bootstrap starting.
I20260927 22:13:43.353317 19106 tablet_bootstrap.cc:492] T dbc1c7400e7d448c90e898d00b448839 P 89f790ec7c04411a87709fa1b3384036: Bootstrap starting.
I20260927 22:13:43.354703 19105 tablet_bootstrap.cc:654] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:43.355341 19107 tablet_bootstrap.cc:492] T dbc1c7400e7d448c90e898d00b448839 P c13485ebe7474ff8b73098316ef381b1: Bootstrap starting.
I20260927 22:13:43.356087 19105 log.cc:824] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:43.357102 19106 tablet_bootstrap.cc:654] T dbc1c7400e7d448c90e898d00b448839 P 89f790ec7c04411a87709fa1b3384036: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:43.358269 19106 log.cc:824] T dbc1c7400e7d448c90e898d00b448839 P 89f790ec7c04411a87709fa1b3384036: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:43.358835 19107 tablet_bootstrap.cc:654] T dbc1c7400e7d448c90e898d00b448839 P c13485ebe7474ff8b73098316ef381b1: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:43.359561 19105 tablet_bootstrap.cc:492] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335: No bootstrap required, opened a new log
I20260927 22:13:43.359907 19107 log.cc:824] T dbc1c7400e7d448c90e898d00b448839 P c13485ebe7474ff8b73098316ef381b1: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:43.359925 19105 ts_tablet_manager.cc:1397] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 22:13:43.362404 19107 tablet_bootstrap.cc:492] T dbc1c7400e7d448c90e898d00b448839 P c13485ebe7474ff8b73098316ef381b1: No bootstrap required, opened a new log
I20260927 22:13:43.362767 19107 ts_tablet_manager.cc:1397] T dbc1c7400e7d448c90e898d00b448839 P c13485ebe7474ff8b73098316ef381b1: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260927 22:13:43.365193 19106 tablet_bootstrap.cc:492] T dbc1c7400e7d448c90e898d00b448839 P 89f790ec7c04411a87709fa1b3384036: No bootstrap required, opened a new log
I20260927 22:13:43.365465 19106 ts_tablet_manager.cc:1397] T dbc1c7400e7d448c90e898d00b448839 P 89f790ec7c04411a87709fa1b3384036: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260927 22:13:43.370420 19107 raft_consensus.cc:348] T dbc1c7400e7d448c90e898d00b448839 P c13485ebe7474ff8b73098316ef381b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c13485ebe7474ff8b73098316ef381b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44349 } } peers { permanent_uuid: "4ec112f0f39b48a1952d472511622335" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39765 } } peers { permanent_uuid: "89f790ec7c04411a87709fa1b3384036" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34027 } }
I20260927 22:13:43.370855 19107 raft_consensus.cc:374] T dbc1c7400e7d448c90e898d00b448839 P c13485ebe7474ff8b73098316ef381b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:43.370963 19107 raft_consensus.cc:729] T dbc1c7400e7d448c90e898d00b448839 P c13485ebe7474ff8b73098316ef381b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c13485ebe7474ff8b73098316ef381b1, State: Initialized, Role: FOLLOWER
I20260927 22:13:43.370921 19105 raft_consensus.cc:348] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c13485ebe7474ff8b73098316ef381b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44349 } } peers { permanent_uuid: "4ec112f0f39b48a1952d472511622335" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39765 } } peers { permanent_uuid: "89f790ec7c04411a87709fa1b3384036" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34027 } }
I20260927 22:13:43.371412 19105 raft_consensus.cc:374] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:43.371513 19105 raft_consensus.cc:729] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec112f0f39b48a1952d472511622335, State: Initialized, Role: FOLLOWER
I20260927 22:13:43.371631 19107 consensus_queue.cc:260] T dbc1c7400e7d448c90e898d00b448839 P c13485ebe7474ff8b73098316ef381b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c13485ebe7474ff8b73098316ef381b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44349 } } peers { permanent_uuid: "4ec112f0f39b48a1952d472511622335" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39765 } } peers { permanent_uuid: "89f790ec7c04411a87709fa1b3384036" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34027 } }
I20260927 22:13:43.372112 19105 consensus_queue.cc:260] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c13485ebe7474ff8b73098316ef381b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44349 } } peers { permanent_uuid: "4ec112f0f39b48a1952d472511622335" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39765 } } peers { permanent_uuid: "89f790ec7c04411a87709fa1b3384036" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34027 } }
I20260927 22:13:43.372785 19087 heartbeater.cc:503] Master 127.0.97.62:44847 was elected leader, sending a full tablet report...
I20260927 22:13:43.373479 19107 ts_tablet_manager.cc:1428] T dbc1c7400e7d448c90e898d00b448839 P c13485ebe7474ff8b73098316ef381b1: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260927 22:13:43.377156 19106 raft_consensus.cc:348] T dbc1c7400e7d448c90e898d00b448839 P 89f790ec7c04411a87709fa1b3384036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c13485ebe7474ff8b73098316ef381b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44349 } } peers { permanent_uuid: "4ec112f0f39b48a1952d472511622335" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39765 } } peers { permanent_uuid: "89f790ec7c04411a87709fa1b3384036" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34027 } }
I20260927 22:13:43.377681 19106 raft_consensus.cc:374] T dbc1c7400e7d448c90e898d00b448839 P 89f790ec7c04411a87709fa1b3384036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:43.377780 19106 raft_consensus.cc:729] T dbc1c7400e7d448c90e898d00b448839 P 89f790ec7c04411a87709fa1b3384036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f790ec7c04411a87709fa1b3384036, State: Initialized, Role: FOLLOWER
I20260927 22:13:43.378690 19106 consensus_queue.cc:260] T dbc1c7400e7d448c90e898d00b448839 P 89f790ec7c04411a87709fa1b3384036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c13485ebe7474ff8b73098316ef381b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44349 } } peers { permanent_uuid: "4ec112f0f39b48a1952d472511622335" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39765 } } peers { permanent_uuid: "89f790ec7c04411a87709fa1b3384036" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34027 } }
I20260927 22:13:43.379733 18953 heartbeater.cc:503] Master 127.0.97.62:44847 was elected leader, sending a full tablet report...
I20260927 22:13:43.380277 19105 ts_tablet_manager.cc:1428] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335: Time spent starting tablet: real 0.020s user 0.020s sys 0.000s
I20260927 22:13:43.381160 19106 ts_tablet_manager.cc:1428] T dbc1c7400e7d448c90e898d00b448839 P 89f790ec7c04411a87709fa1b3384036: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260927 22:13:43.434475 19112 raft_consensus.cc:482] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:13:43.434892 19112 raft_consensus.cc:504] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c13485ebe7474ff8b73098316ef381b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44349 } } peers { permanent_uuid: "4ec112f0f39b48a1952d472511622335" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39765 } } peers { permanent_uuid: "89f790ec7c04411a87709fa1b3384036" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34027 } }
I20260927 22:13:43.436656 19112 leader_election.cc:288] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c13485ebe7474ff8b73098316ef381b1 (127.0.97.3:44349), 89f790ec7c04411a87709fa1b3384036 (127.0.97.2:34027)
W20260927 22:13:43.442370 18820 tablet.cc:2369] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:13:43.444001 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:43.437369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60585 (local address 127.0.97.2:34027)
0927 22:13:43.437620 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:43.437629 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:43.438327 (+ 698us) server_negotiation.cc:407] Connection header received
0927 22:13:43.438493 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:43.438497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:43.438588 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:43.438689 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:43.439342 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.440096 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.441872 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.442881 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.443013 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:43.443108 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:43.443617 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 22:13:43.443740 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:13:43.444118 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:43.437085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34027 (local address 127.0.97.1:60585)
0927 22:13:43.438201 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:43.438220 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:43.438370 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:43.438702 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:43.438706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:43.438730 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:43.439165 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:43.439178 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.440231 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.440237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:43.441710 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:43.441722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.443255 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.443261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:43.443435 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:43.443477 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 22:13:43.443593 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 22:13:43.443723 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"mutex_wait_us":5,"thread_start_us":833,"threads_started":1}
I20260927 22:13:43.444221 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:43.436984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52375 (local address 127.0.97.3:44349)
0927 22:13:43.437233 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:43.437243 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:43.438192 (+ 949us) server_negotiation.cc:407] Connection header received
0927 22:13:43.438397 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:43.438405 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:43.438492 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:43.438581 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:43.439348 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.440232 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.441951 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.443131 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.443255 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:43.443329 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:43.443836 (+ 507us) server_negotiation.cc:299] Negotiation successful
0927 22:13:43.443971 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:13:43.444962 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:43.436876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44349 (local address 127.0.97.1:52375)
0927 22:13:43.438060 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:43.438082 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:43.438232 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:43.438639 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:43.438646 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:43.438676 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:43.439188 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:43.439200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.440365 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.440368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:43.441805 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:43.441822 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.443480 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.443485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:43.443645 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:43.443685 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 22:13:43.444483 (+ 798us) client_negotiation.cc:240] Negotiation successful
0927 22:13:43.444619 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1030,"mutex_wait_us":19,"thread_start_us":781,"threads_started":1}
I20260927 22:13:43.445147 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc1c7400e7d448c90e898d00b448839" candidate_uuid: "4ec112f0f39b48a1952d472511622335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f790ec7c04411a87709fa1b3384036" is_pre_election: true
I20260927 22:13:43.445667 18908 raft_consensus.cc:2454] T dbc1c7400e7d448c90e898d00b448839 P 89f790ec7c04411a87709fa1b3384036 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ec112f0f39b48a1952d472511622335 in term 0.
I20260927 22:13:43.445789 19042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc1c7400e7d448c90e898d00b448839" candidate_uuid: "4ec112f0f39b48a1952d472511622335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c13485ebe7474ff8b73098316ef381b1" is_pre_election: true
I20260927 22:13:43.446303 19042 raft_consensus.cc:2454] T dbc1c7400e7d448c90e898d00b448839 P c13485ebe7474ff8b73098316ef381b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ec112f0f39b48a1952d472511622335 in term 0.
I20260927 22:13:43.446337 18710 leader_election.cc:302] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ec112f0f39b48a1952d472511622335, 89f790ec7c04411a87709fa1b3384036; no voters:
I20260927 22:13:43.446765 19112 raft_consensus.cc:2781] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:13:43.446925 19112 raft_consensus.cc:482] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:13:43.447041 19112 raft_consensus.cc:3037] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:43.449227 19112 raft_consensus.cc:504] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c13485ebe7474ff8b73098316ef381b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44349 } } peers { permanent_uuid: "4ec112f0f39b48a1952d472511622335" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39765 } } peers { permanent_uuid: "89f790ec7c04411a87709fa1b3384036" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34027 } }
I20260927 22:13:43.449816 19112 leader_election.cc:288] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335 [CANDIDATE]: Term 1 election: Requested vote from peers c13485ebe7474ff8b73098316ef381b1 (127.0.97.3:44349), 89f790ec7c04411a87709fa1b3384036 (127.0.97.2:34027)
I20260927 22:13:43.450290 19042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc1c7400e7d448c90e898d00b448839" candidate_uuid: "4ec112f0f39b48a1952d472511622335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c13485ebe7474ff8b73098316ef381b1"
I20260927 22:13:43.450309 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc1c7400e7d448c90e898d00b448839" candidate_uuid: "4ec112f0f39b48a1952d472511622335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f790ec7c04411a87709fa1b3384036"
I20260927 22:13:43.450498 18908 raft_consensus.cc:3037] T dbc1c7400e7d448c90e898d00b448839 P 89f790ec7c04411a87709fa1b3384036 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:43.450491 19042 raft_consensus.cc:3037] T dbc1c7400e7d448c90e898d00b448839 P c13485ebe7474ff8b73098316ef381b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:43.452651 18908 raft_consensus.cc:2454] T dbc1c7400e7d448c90e898d00b448839 P 89f790ec7c04411a87709fa1b3384036 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ec112f0f39b48a1952d472511622335 in term 1.
I20260927 22:13:43.452921 19042 raft_consensus.cc:2454] T dbc1c7400e7d448c90e898d00b448839 P c13485ebe7474ff8b73098316ef381b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ec112f0f39b48a1952d472511622335 in term 1.
I20260927 22:13:43.452994 18710 leader_election.cc:302] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ec112f0f39b48a1952d472511622335, 89f790ec7c04411a87709fa1b3384036; no voters:
I20260927 22:13:43.453378 19112 raft_consensus.cc:2781] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:13:43.453895 19112 raft_consensus.cc:686] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335 [term 1 LEADER]: Becoming Leader. State: Replica: 4ec112f0f39b48a1952d472511622335, State: Running, Role: LEADER
I20260927 22:13:43.454391 19112 consensus_queue.cc:237] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c13485ebe7474ff8b73098316ef381b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44349 } } peers { permanent_uuid: "4ec112f0f39b48a1952d472511622335" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39765 } } peers { permanent_uuid: "89f790ec7c04411a87709fa1b3384036" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34027 } }
I20260927 22:13:43.459421 18629 catalog_manager.cc:5495] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ec112f0f39b48a1952d472511622335 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "4ec112f0f39b48a1952d472511622335" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c13485ebe7474ff8b73098316ef381b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ec112f0f39b48a1952d472511622335" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89f790ec7c04411a87709fa1b3384036" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34027 } health_report { overall_health: UNKNOWN } } }
W20260927 22:13:43.486352 18954 tablet.cc:2369] T dbc1c7400e7d448c90e898d00b448839 P 89f790ec7c04411a87709fa1b3384036: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:13:43.486900 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:43.481232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53640 (local address 127.0.97.1:39765)
0927 22:13:43.481440 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:43.481448 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:43.481468 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:13:43.481721 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:43.481725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:43.481816 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:43.481982 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:13:43.482371 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.483158 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.484539 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.484948 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.485088 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:43.485680 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:13:43.486198 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:13:43.486394 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:43.486541 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 22:13:43.486610 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:13:43.495378 18908 raft_consensus.cc:1261] T dbc1c7400e7d448c90e898d00b448839 P 89f790ec7c04411a87709fa1b3384036 [term 1 FOLLOWER]: Refusing update from remote peer 4ec112f0f39b48a1952d472511622335: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:13:43.495401 19042 raft_consensus.cc:1261] T dbc1c7400e7d448c90e898d00b448839 P c13485ebe7474ff8b73098316ef381b1 [term 1 FOLLOWER]: Refusing update from remote peer 4ec112f0f39b48a1952d472511622335: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:13:43.496376 19112 consensus_queue.cc:1035] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f790ec7c04411a87709fa1b3384036" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:13:43.496882 19116 consensus_queue.cc:1035] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c13485ebe7474ff8b73098316ef381b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:13:43.497280 19088 tablet.cc:2369] T dbc1c7400e7d448c90e898d00b448839 P c13485ebe7474ff8b73098316ef381b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:13:43.527281 18629 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:13:43.527562 18629 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:13:43.531102 18629 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0169298caea4ac7a24c8adb416269ab: Sending DeleteTablet for 3 replicas of tablet dbc1c7400e7d448c90e898d00b448839
I20260927 22:13:43.532300 19022 tablet_service.cc:1514] Processing DeleteTablet for tablet dbc1c7400e7d448c90e898d00b448839 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:13:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46584
I20260927 22:13:43.532316 18754 tablet_service.cc:1514] Processing DeleteTablet for tablet dbc1c7400e7d448c90e898d00b448839 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:13:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53638
I20260927 22:13:43.532310 18888 tablet_service.cc:1514] Processing DeleteTablet for tablet dbc1c7400e7d448c90e898d00b448839 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:13:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46478
I20260927 22:13:43.541241 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 18691
I20260927 22:13:43.546449 19130 tablet_replica.cc:330] T dbc1c7400e7d448c90e898d00b448839 P 89f790ec7c04411a87709fa1b3384036: stopping tablet replica
I20260927 22:13:43.547004 19130 raft_consensus.cc:2229] T dbc1c7400e7d448c90e898d00b448839 P 89f790ec7c04411a87709fa1b3384036 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:13:43.547734 19130 raft_consensus.cc:2258] T dbc1c7400e7d448c90e898d00b448839 P 89f790ec7c04411a87709fa1b3384036 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:13:43.548691 19131 tablet_replica.cc:330] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335: stopping tablet replica
I20260927 22:13:43.549129 19131 raft_consensus.cc:2229] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:13:43.549923 19131 raft_consensus.cc:2258] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:13:43.552418 19131 ts_tablet_manager.cc:1905] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:13:43.555595 19130 ts_tablet_manager.cc:1905] T dbc1c7400e7d448c90e898d00b448839 P 89f790ec7c04411a87709fa1b3384036: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:13:43.556181 19129 tablet_replica.cc:330] T dbc1c7400e7d448c90e898d00b448839 P c13485ebe7474ff8b73098316ef381b1: stopping tablet replica
I20260927 22:13:43.556669 19129 raft_consensus.cc:2229] T dbc1c7400e7d448c90e898d00b448839 P c13485ebe7474ff8b73098316ef381b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:13:43.558282 19129 raft_consensus.cc:2258] T dbc1c7400e7d448c90e898d00b448839 P c13485ebe7474ff8b73098316ef381b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:13:43.560151 19131 ts_tablet_manager.cc:1918] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:13:43.560391 19131 log.cc:1196] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-0/wal/wals/dbc1c7400e7d448c90e898d00b448839
I20260927 22:13:43.560946 19131 ts_tablet_manager.cc:1939] T dbc1c7400e7d448c90e898d00b448839 P 4ec112f0f39b48a1952d472511622335: Deleting consensus metadata
Sep 27 22:13:43 dist-test-slave-mp03 krb5kdc[18585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:13:43.562647 19129 ts_tablet_manager.cc:1905] T dbc1c7400e7d448c90e898d00b448839 P c13485ebe7474ff8b73098316ef381b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:13:43.562825 18616 catalog_manager.cc:4841] TS 4ec112f0f39b48a1952d472511622335 (127.0.97.1:39765): tablet dbc1c7400e7d448c90e898d00b448839 (table test-table [id=855518b8cc4746509018446dba02618d]) successfully deleted
I20260927 22:13:43.568567 19129 ts_tablet_manager.cc:1918] T dbc1c7400e7d448c90e898d00b448839 P c13485ebe7474ff8b73098316ef381b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:13:43.568817 19129 log.cc:1196] T dbc1c7400e7d448c90e898d00b448839 P c13485ebe7474ff8b73098316ef381b1: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/dbc1c7400e7d448c90e898d00b448839
I20260927 22:13:43.569427 19129 ts_tablet_manager.cc:1939] T dbc1c7400e7d448c90e898d00b448839 P c13485ebe7474ff8b73098316ef381b1: Deleting consensus metadata
I20260927 22:13:43.570482 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:43.542961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53646 (local address 127.0.97.1:39765)
0927 22:13:43.543185 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:43.543192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:43.546077 (+ 2885us) server_negotiation.cc:407] Connection header received
0927 22:13:43.546160 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:43.546167 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:43.546252 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:43.546359 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:43.550400 (+ 4041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.551622 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.556077 (+ 4455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.556830 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.556939 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:43.562607 (+ 5668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:43.562622 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:43.562640 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:43.562681 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:43.566844 (+ 4163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:43.567798 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:43.567808 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:43.567812 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:43.567981 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:43.568459 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:43.568466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:43.568470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:43.568940 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:43.569241 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:43.569827 (+ 586us) server_negotiation.cc:299] Negotiation successful
0927 22:13:43.570094 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:13:43.570905 19130 ts_tablet_manager.cc:1918] T dbc1c7400e7d448c90e898d00b448839 P 89f790ec7c04411a87709fa1b3384036: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:13:43.571076 19130 log.cc:1196] T dbc1c7400e7d448c90e898d00b448839 P 89f790ec7c04411a87709fa1b3384036: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/dbc1c7400e7d448c90e898d00b448839
I20260927 22:13:43.571637 19130 ts_tablet_manager.cc:1939] T dbc1c7400e7d448c90e898d00b448839 P 89f790ec7c04411a87709fa1b3384036: Deleting consensus metadata
I20260927 22:13:43.571936 18814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:43.572834 18615 catalog_manager.cc:4841] TS c13485ebe7474ff8b73098316ef381b1 (127.0.97.3:44349): tablet dbc1c7400e7d448c90e898d00b448839 (table test-table [id=855518b8cc4746509018446dba02618d]) successfully deleted
I20260927 22:13:43.573566 18617 catalog_manager.cc:4841] TS 89f790ec7c04411a87709fa1b3384036 (127.0.97.2:34027): tablet dbc1c7400e7d448c90e898d00b448839 (table test-table [id=855518b8cc4746509018446dba02618d]) successfully deleted
I20260927 22:13:43.676338 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 18691
I20260927 22:13:43.695024 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 18825
Sep 27 22:13:43 dist-test-slave-mp03 krb5kdc[18585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:13:43.709551 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:43.695821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46484 (local address 127.0.97.2:34027)
0927 22:13:43.696004 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:43.696008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:43.696026 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:13:43.696152 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:43.696156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:43.696217 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:43.696320 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:43.697479 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.698197 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.699514 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.699882 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.699969 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:43.704228 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:43.704240 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:43.704255 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:43.704301 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:43.707080 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:43.707744 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:43.707750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:43.707753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:43.707868 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:43.708173 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:43.708176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:43.708178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:43.708526 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:43.708737 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:43.709079 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 22:13:43.709306 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:13:43.710249 18948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:43.799454 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 18825
I20260927 22:13:43.817440 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 18959
Sep 27 22:13:43 dist-test-slave-mp03 krb5kdc[18585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:13:43.833617 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:43.818234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46596 (local address 127.0.97.3:44349)
0927 22:13:43.818408 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:43.818413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:43.818431 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:13:43.818557 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:43.818560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:43.818694 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:43.818782 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:43.820153 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.820868 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.822295 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:43.822840 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:43.822988 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:43.827496 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:43.827511 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:43.827527 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:43.827567 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:43.830697 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:43.831466 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:43.831482 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:43.831485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:43.831637 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:43.832027 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:43.832033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:43.832036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:43.832426 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:43.832719 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:43.833051 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 22:13:43.833301 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:13:43.834249 19082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:43.926113 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 18959
I20260927 22:13:43.942817 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 18598
I20260927 22:13:43.943909 18660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:44.055269 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 18598
2026-09-27T22:13:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4287 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T22:13:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:13:44Z Disabled control of system clock
I20260927 22:13:44.135511 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43637
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:13:44.473042 19140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:44.473367 19140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:44.473462 19140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:44.480887 19140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:13:44.481001 19140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:44.481038 19140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:44.481096 19140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:13:44.481139 19140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:13:44.490643 19140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43637
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:44.491927 19140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:44.493682 19140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:44.505653 19148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:44.505666 19146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:44.505674 19145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:44.506740 19140 server_base.cc:1034] running on GCE node
I20260927 22:13:44.507530 19140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:44.508944 19140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:44.510150 19140 hybrid_clock.cc:648] HybridClock initialized: now 1790547224510103 us; error 46 us; skew 500 ppm
I20260927 22:13:44.510475 19140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:13:44.512795 19140 webserver.cc:460] Webserver started at http://127.0.97.62:45029/ using document root <none> and password file <none>
I20260927 22:13:44.513537 19140 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:44.513625 19140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:44.513895 19140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:44.516851 19140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "4be380cd3add4229857dab88980174dc"
format_stamp: "Formatted at 2026-09-27 22:13:44 on dist-test-slave-mp03"
server_key: "da091680b9a32510ccbd24c054b8e006"
server_key_iv: "5c472b4f6fd368b84198003d68e9b7d8"
server_key_version: "encryptionkey@0"
I20260927 22:13:44.517635 19140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "4be380cd3add4229857dab88980174dc"
format_stamp: "Formatted at 2026-09-27 22:13:44 on dist-test-slave-mp03"
server_key: "da091680b9a32510ccbd24c054b8e006"
server_key_iv: "5c472b4f6fd368b84198003d68e9b7d8"
server_key_version: "encryptionkey@0"
I20260927 22:13:44.523007 19140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 22:13:44.526428 19154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:44.527936 19140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:13:44.528118 19140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "4be380cd3add4229857dab88980174dc"
format_stamp: "Formatted at 2026-09-27 22:13:44 on dist-test-slave-mp03"
server_key: "da091680b9a32510ccbd24c054b8e006"
server_key_iv: "5c472b4f6fd368b84198003d68e9b7d8"
server_key_version: "encryptionkey@0"
I20260927 22:13:44.528344 19140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:44.544512 19140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:44.545348 19140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:44.545635 19140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:44.565218 19140 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43637
I20260927 22:13:44.565254 19205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43637 every 8 connection(s)
I20260927 22:13:44.566624 19140 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:13:44.568068 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 19140
I20260927 22:13:44.568255 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:13:44.568796 388 external_mini_cluster.cc:1428] Setting key f0233caa93890f3ae6970eea7e92ca2c
I20260927 22:13:44.571313 19206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:44.582921 19209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:44.570809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56572 (local address 127.0.97.62:43637)
0927 22:13:44.571834 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:44.571844 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:44.571886 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:13:44.573499 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:44.573520 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:44.574010 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:44.574358 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:44.574957 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:44.576332 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:44.578291 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:44.579081 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:44.579219 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:44.580469 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:44.580476 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:44.580489 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:44.581069 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:44.581153 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:44.581178 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:44.581278 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:44.581511 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 22:13:44.581911 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":376,"threads_started":1}
I20260927 22:13:44.584093 19206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4be380cd3add4229857dab88980174dc: Bootstrap starting.
I20260927 22:13:44.587628 19206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4be380cd3add4229857dab88980174dc: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:44.588760 19206 log.cc:824] T 00000000000000000000000000000000 P 4be380cd3add4229857dab88980174dc: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:44.591571 19206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4be380cd3add4229857dab88980174dc: No bootstrap required, opened a new log
I20260927 22:13:44.599531 19206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4be380cd3add4229857dab88980174dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be380cd3add4229857dab88980174dc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43637 } }
I20260927 22:13:44.599929 19206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4be380cd3add4229857dab88980174dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:44.600057 19206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4be380cd3add4229857dab88980174dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4be380cd3add4229857dab88980174dc, State: Initialized, Role: FOLLOWER
I20260927 22:13:44.600625 19206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4be380cd3add4229857dab88980174dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be380cd3add4229857dab88980174dc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43637 } }
I20260927 22:13:44.600862 19206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4be380cd3add4229857dab88980174dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:13:44.601015 19206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4be380cd3add4229857dab88980174dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:13:44.601215 19206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4be380cd3add4229857dab88980174dc [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:44.603649 19206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4be380cd3add4229857dab88980174dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be380cd3add4229857dab88980174dc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43637 } }
I20260927 22:13:44.604189 19206 leader_election.cc:302] T 00000000000000000000000000000000 P 4be380cd3add4229857dab88980174dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4be380cd3add4229857dab88980174dc; no voters:
I20260927 22:13:44.604773 19206 leader_election.cc:288] T 00000000000000000000000000000000 P 4be380cd3add4229857dab88980174dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:13:44.604873 19211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4be380cd3add4229857dab88980174dc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:13:44.605639 19211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4be380cd3add4229857dab88980174dc [term 1 LEADER]: Becoming Leader. State: Replica: 4be380cd3add4229857dab88980174dc, State: Running, Role: LEADER
I20260927 22:13:44.606112 19211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4be380cd3add4229857dab88980174dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be380cd3add4229857dab88980174dc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43637 } }
I20260927 22:13:44.606995 19206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4be380cd3add4229857dab88980174dc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:13:44.608985 19213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4be380cd3add4229857dab88980174dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4be380cd3add4229857dab88980174dc. Latest consensus state: current_term: 1 leader_uuid: "4be380cd3add4229857dab88980174dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be380cd3add4229857dab88980174dc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43637 } } }
I20260927 22:13:44.609407 19213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4be380cd3add4229857dab88980174dc [sys.catalog]: This master's current role is: LEADER
I20260927 22:13:44.610291 19219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:13:44.610150 19212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4be380cd3add4229857dab88980174dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4be380cd3add4229857dab88980174dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be380cd3add4229857dab88980174dc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43637 } } }
I20260927 22:13:44.610461 19212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4be380cd3add4229857dab88980174dc [sys.catalog]: This master's current role is: LEADER
I20260927 22:13:44.618112 19219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:13:44.627110 19219 catalog_manager.cc:1347] Generated new cluster ID: 0075a9e7a7304635b40f4be90429c6a6
I20260927 22:13:44.627238 19219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:13:44.646957 19219 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:13:44.648307 19219 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:13:44.655685 19219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4be380cd3add4229857dab88980174dc: Generated new TSK 0
I20260927 22:13:44.656554 19219 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:13:44.667059 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43637
--builtin_ntp_servers=127.0.97.20:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:13:45.010501 19230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:45.010874 19230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:45.010986 19230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:45.018469 19230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:45.018566 19230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:45.018756 19230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:13:45.027853 19230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:45.029104 19230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:45.030906 19230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:45.044221 19235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:45.044204 19236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:45.044483 19238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:45.045624 19230 server_base.cc:1034] running on GCE node
I20260927 22:13:45.046257 19230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:45.047365 19230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:45.048579 19230 hybrid_clock.cc:648] HybridClock initialized: now 1790547225048536 us; error 30 us; skew 500 ppm
I20260927 22:13:45.048993 19230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:13:45.051328 19230 webserver.cc:460] Webserver started at http://127.0.97.1:39109/ using document root <none> and password file <none>
I20260927 22:13:45.052018 19230 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:45.052148 19230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:45.052471 19230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:45.055327 19230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "1cfd364dfdb6468a87a4e0b1507e250b"
format_stamp: "Formatted at 2026-09-27 22:13:45 on dist-test-slave-mp03"
server_key: "fcef0dab2e40e28bf5e618e2b5344f7b"
server_key_iv: "2a9af67de0523d2c55045555b59e3871"
server_key_version: "encryptionkey@0"
I20260927 22:13:45.056133 19230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1cfd364dfdb6468a87a4e0b1507e250b"
format_stamp: "Formatted at 2026-09-27 22:13:45 on dist-test-slave-mp03"
server_key: "fcef0dab2e40e28bf5e618e2b5344f7b"
server_key_iv: "2a9af67de0523d2c55045555b59e3871"
server_key_version: "encryptionkey@0"
I20260927 22:13:45.061480 19230 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:13:45.065056 19244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:45.066404 19230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:13:45.066680 19230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "1cfd364dfdb6468a87a4e0b1507e250b"
format_stamp: "Formatted at 2026-09-27 22:13:45 on dist-test-slave-mp03"
server_key: "fcef0dab2e40e28bf5e618e2b5344f7b"
server_key_iv: "2a9af67de0523d2c55045555b59e3871"
server_key_version: "encryptionkey@0"
I20260927 22:13:45.066963 19230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:45.088727 19230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:45.089624 19230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:45.089924 19230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:45.090977 19230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:13:45.092447 19230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:13:45.092573 19230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:45.092695 19230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:13:45.092788 19230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:45.126187 19230 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38599
I20260927 22:13:45.126210 19356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38599 every 8 connection(s)
I20260927 22:13:45.127617 19230 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:13:45.136140 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 19230
I20260927 22:13:45.136327 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:13:45.136745 388 external_mini_cluster.cc:1428] Setting key d6c52781046ac8a1dfcc32c89f1e6551
I20260927 22:13:45.139494 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43637
--builtin_ntp_servers=127.0.97.20:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:13:45.140576 19359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:45.130093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49653 (local address 127.0.97.62:43637)
0927 22:13:45.130555 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:45.130559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:45.131721 (+ 1162us) server_negotiation.cc:407] Connection header received
0927 22:13:45.133702 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:45.133707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:45.133802 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:45.133929 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:45.135027 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:45.135737 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:45.137484 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:45.137962 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:45.138065 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:45.139597 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:45.139605 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:45.139613 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:45.139637 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:45.139691 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:45.139711 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:45.139816 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:45.140022 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 22:13:45.140129 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":352,"thread_start_us":252,"threads_started":1}
I20260927 22:13:45.141264 19360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:45.130651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43637 (local address 127.0.97.1:49653)
0927 22:13:45.131576 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:45.131617 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:45.133438 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:45.134149 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:45.134157 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:45.134239 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:13:45.134863 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:45.134882 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:45.135888 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:45.135895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:45.137330 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:45.137345 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:45.138184 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:45.138191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:45.138537 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:45.139284 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:13:45.139287 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:13:45.139330 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:13:45.139407 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:13:45.139813 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:13:45.139823 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:13:45.139829 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 22:13:45.139962 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 22:13:45.140235 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":250,"threads_started":1}
I20260927 22:13:45.142779 19357 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43637
I20260927 22:13:45.143074 19357 heartbeater.cc:464] Registering TS with master...
I20260927 22:13:45.143801 19357 heartbeater.cc:511] Master 127.0.97.62:43637 requested a full tablet report, sending...
I20260927 22:13:45.145748 19171 ts_manager.cc:194] Registered new tserver with Master: 1cfd364dfdb6468a87a4e0b1507e250b (127.0.97.1:38599)
I20260927 22:13:45.147325 19171 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:49653
W20260927 22:13:45.469707 19361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:45.470037 19361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:45.470211 19361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:45.477579 19361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:45.477741 19361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:45.477926 19361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:13:45.487200 19361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:45.488528 19361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:45.490474 19361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:45.503829 19366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:45.503851 19367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:45.503914 19369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:45.504493 19361 server_base.cc:1034] running on GCE node
I20260927 22:13:45.505137 19361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:45.506079 19361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:45.507285 19361 hybrid_clock.cc:648] HybridClock initialized: now 1790547225507248 us; error 35 us; skew 500 ppm
I20260927 22:13:45.507663 19361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:13:45.510181 19361 webserver.cc:460] Webserver started at http://127.0.97.2:46619/ using document root <none> and password file <none>
I20260927 22:13:45.510993 19361 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:45.511140 19361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:45.511473 19361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:45.514402 19361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "36799e8cea1a443491b1d6f0d0775f8f"
format_stamp: "Formatted at 2026-09-27 22:13:45 on dist-test-slave-mp03"
server_key: "fda031d2d805e1867a00f4cba4150f4c"
server_key_iv: "6d10101779a6261158b6cec424cc02a8"
server_key_version: "encryptionkey@0"
I20260927 22:13:45.515283 19361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "36799e8cea1a443491b1d6f0d0775f8f"
format_stamp: "Formatted at 2026-09-27 22:13:45 on dist-test-slave-mp03"
server_key: "fda031d2d805e1867a00f4cba4150f4c"
server_key_iv: "6d10101779a6261158b6cec424cc02a8"
server_key_version: "encryptionkey@0"
I20260927 22:13:45.520349 19361 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 22:13:45.524062 19375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:45.525490 19361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:13:45.525651 19361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "36799e8cea1a443491b1d6f0d0775f8f"
format_stamp: "Formatted at 2026-09-27 22:13:45 on dist-test-slave-mp03"
server_key: "fda031d2d805e1867a00f4cba4150f4c"
server_key_iv: "6d10101779a6261158b6cec424cc02a8"
server_key_version: "encryptionkey@0"
I20260927 22:13:45.525897 19361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:45.564455 19361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:45.565331 19361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:45.565623 19361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:45.566685 19361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:13:45.568181 19361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:13:45.568302 19361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:45.568431 19361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:13:45.568526 19361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:45.609104 19361 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43271
I20260927 22:13:45.609203 19487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43271 every 8 connection(s)
I20260927 22:13:45.610437 19361 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:13:45.614534 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 19361
I20260927 22:13:45.614717 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:13:45.615278 388 external_mini_cluster.cc:1428] Setting key d78a1bf8f22fcbac502adee18e3f2566
I20260927 22:13:45.618330 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43637
--builtin_ntp_servers=127.0.97.20:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:13:45.625422 19359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:45.613212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43179 (local address 127.0.97.62:43637)
0927 22:13:45.613424 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:45.613431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:45.615139 (+ 1708us) server_negotiation.cc:407] Connection header received
0927 22:13:45.617351 (+ 2212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:45.617355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:45.617418 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:45.617499 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:45.619007 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:45.619811 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:45.622356 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:45.622868 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:45.622994 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:45.624698 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:45.624702 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:45.624706 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:45.624726 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:45.624758 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:45.624770 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:45.624843 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:45.625100 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 22:13:45.625182 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:13:45.627048 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:45.613806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43637 (local address 127.0.97.2:43179)
0927 22:13:45.614963 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:45.615017 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:45.617068 (+ 2051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:45.617767 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:45.617778 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:45.617899 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:13:45.618770 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:45.618799 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:45.619975 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:45.619983 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:45.622161 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:45.622178 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:45.623113 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:45.623122 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:45.623493 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:45.624300 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:13:45.624306 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:13:45.624370 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:13:45.624487 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:13:45.624870 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:13:45.624883 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:13:45.624893 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 22:13:45.625086 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 22:13:45.625554 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":328,"threads_started":1}
I20260927 22:13:45.628578 19488 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43637
I20260927 22:13:45.628917 19488 heartbeater.cc:464] Registering TS with master...
I20260927 22:13:45.629854 19488 heartbeater.cc:511] Master 127.0.97.62:43637 requested a full tablet report, sending...
I20260927 22:13:45.631449 19171 ts_manager.cc:194] Registered new tserver with Master: 36799e8cea1a443491b1d6f0d0775f8f (127.0.97.2:43271)
I20260927 22:13:45.632345 19171 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:43179
W20260927 22:13:45.953800 19491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:45.954097 19491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:45.954221 19491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:45.961722 19491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:45.961833 19491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:45.961942 19491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:13:45.971493 19491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:45.972851 19491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:45.974627 19491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:45.987895 19496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:45.988106 19499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:45.987916 19497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:45.988718 19491 server_base.cc:1034] running on GCE node
I20260927 22:13:45.989408 19491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:45.990347 19491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:45.991585 19491 hybrid_clock.cc:648] HybridClock initialized: now 1790547225991516 us; error 77 us; skew 500 ppm
I20260927 22:13:45.991920 19491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:13:45.994247 19491 webserver.cc:460] Webserver started at http://127.0.97.3:33529/ using document root <none> and password file <none>
I20260927 22:13:45.995081 19491 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:45.995218 19491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:45.995553 19491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:45.998535 19491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "493f9c4e99ce43dcac9823ba9fa28f7e"
format_stamp: "Formatted at 2026-09-27 22:13:45 on dist-test-slave-mp03"
server_key: "3ff3d45d6071883ea04af73a9ebc4fb2"
server_key_iv: "a0d21123e7f685fe8629f48f33808e85"
server_key_version: "encryptionkey@0"
I20260927 22:13:45.999441 19491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "493f9c4e99ce43dcac9823ba9fa28f7e"
format_stamp: "Formatted at 2026-09-27 22:13:45 on dist-test-slave-mp03"
server_key: "3ff3d45d6071883ea04af73a9ebc4fb2"
server_key_iv: "a0d21123e7f685fe8629f48f33808e85"
server_key_version: "encryptionkey@0"
I20260927 22:13:46.004626 19491 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 22:13:46.008220 19505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:46.009660 19491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:13:46.009979 19491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "493f9c4e99ce43dcac9823ba9fa28f7e"
format_stamp: "Formatted at 2026-09-27 22:13:45 on dist-test-slave-mp03"
server_key: "3ff3d45d6071883ea04af73a9ebc4fb2"
server_key_iv: "a0d21123e7f685fe8629f48f33808e85"
server_key_version: "encryptionkey@0"
I20260927 22:13:46.010205 19491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:46.025911 19491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:46.026863 19491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:46.027144 19491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:46.028134 19491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:13:46.029695 19491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:13:46.029834 19491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:46.029959 19491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:13:46.030045 19491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:46.063424 19491 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43395
I20260927 22:13:46.063519 19617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43395 every 8 connection(s)
I20260927 22:13:46.064829 19491 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:13:46.066617 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 19491
I20260927 22:13:46.066785 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:13:46.067284 388 external_mini_cluster.cc:1428] Setting key 15d9fe774a5ba2148a60dd10b4966598
I20260927 22:13:46.077906 19359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:46.068713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35633 (local address 127.0.97.62:43637)
0927 22:13:46.068920 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:46.068926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:46.069497 (+ 571us) server_negotiation.cc:407] Connection header received
0927 22:13:46.071658 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:46.071663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:46.071775 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:46.071862 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:46.072900 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.073645 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.075326 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.075767 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.075875 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:46.077162 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:46.077167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:46.077170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:46.077188 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:46.077222 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:46.077235 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:46.077313 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:46.077542 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 22:13:46.077629 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:13:46.078956 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:46.068295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43637 (local address 127.0.97.3:35633)
0927 22:13:46.069330 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:46.069385 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:46.071407 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:46.072065 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:46.072072 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:46.072159 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:13:46.072744 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:46.072760 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.073771 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.073774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:46.075186 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:46.075199 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.075972 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.075977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:46.076187 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:46.076854 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:13:46.076857 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:13:46.076896 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:13:46.076975 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:13:46.077331 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:13:46.077354 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:13:46.077360 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 22:13:46.077510 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 22:13:46.077814 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":397,"threads_started":1}
I20260927 22:13:46.080355 19618 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43637
I20260927 22:13:46.080662 19618 heartbeater.cc:464] Registering TS with master...
I20260927 22:13:46.081424 19618 heartbeater.cc:511] Master 127.0.97.62:43637 requested a full tablet report, sending...
I20260927 22:13:46.082937 19170 ts_manager.cc:194] Registered new tserver with Master: 493f9c4e99ce43dcac9823ba9fa28f7e (127.0.97.3:43395)
I20260927 22:13:46.083802 19170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:35633
I20260927 22:13:46.095188 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:13:46.105088 19359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:46.099296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56580 (local address 127.0.97.62:43637)
0927 22:13:46.099523 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:46.099531 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:46.099868 (+ 337us) server_negotiation.cc:407] Connection header received
0927 22:13:46.100039 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:46.100043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:46.100120 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:46.100241 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:46.100649 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.101371 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.102747 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.103218 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.103317 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:46.104362 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:46.104367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:46.104370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:46.104386 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:46.104416 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:46.104427 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:46.104537 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:46.104705 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 22:13:46.104782 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:13:46.109103 19170 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:13:46.116367 19170 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:13:46.150111 19357 heartbeater.cc:503] Master 127.0.97.62:43637 was elected leader, sending a full tablet report...
I20260927 22:13:46.150244 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:46.129093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38599 (local address 127.0.0.1:41614)
0927 22:13:46.140950 (+ 11857us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:46.140973 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:46.141174 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:46.142165 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:46.142170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:46.142229 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:46.142907 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:46.142921 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.144920 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.144928 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:46.147308 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:46.147321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.149176 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.149183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:46.149382 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:46.149440 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 22:13:46.149573 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 22:13:46.149803 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11718,"mutex_wait_us":134,"spinlock_wait_cycles":896,"thread_start_us":12835,"threads_started":1}
I20260927 22:13:46.150480 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:46.129588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41614 (local address 127.0.97.1:38599)
0927 22:13:46.140597 (+ 11009us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:46.140609 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:46.141072 (+ 463us) server_negotiation.cc:407] Connection header received
0927 22:13:46.141382 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:46.141394 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:46.141684 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:46.141857 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:46.143139 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.144733 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.147511 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.148974 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.149113 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:46.149412 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:46.149656 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 22:13:46.149848 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10797,"thread_start_us":10693,"threads_started":1}
I20260927 22:13:46.151460 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:46.128821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43271 (local address 127.0.0.1:36336)
0927 22:13:46.138822 (+ 10001us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:46.138881 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:46.139048 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:46.142082 (+ 3034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:46.142090 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:46.142118 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:46.142885 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:46.142900 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.144873 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.144878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:46.147592 (+ 2714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:46.147609 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.149615 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.149626 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:46.150824 (+ 1198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:46.150875 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 22:13:46.150948 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 22:13:46.151076 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9705,"mutex_wait_us":153,"spinlock_wait_cycles":38656,"thread_start_us":11814,"threads_started":1}
I20260927 22:13:46.151695 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:46.129008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36336 (local address 127.0.97.2:43271)
0927 22:13:46.141308 (+ 12300us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:46.141319 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:46.141354 (+ 35us) server_negotiation.cc:407] Connection header received
0927 22:13:46.141464 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:46.141474 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:46.141756 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:46.142924 (+ 1168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:46.143158 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.144739 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.147792 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.149125 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.149255 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:46.149463 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:46.151218 (+ 1755us) server_negotiation.cc:299] Negotiation successful
0927 22:13:46.151379 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12106,"thread_start_us":12621,"threads_started":1}
I20260927 22:13:46.152199 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:46.132115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45876 (local address 127.0.97.3:43395)
0927 22:13:46.143081 (+ 10966us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:46.143089 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:46.143122 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:13:46.143248 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:46.143253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:46.143420 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:46.143604 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:46.144164 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.145416 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.148168 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.149090 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.149243 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:46.149505 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:46.150332 (+ 827us) server_negotiation.cc:299] Negotiation successful
0927 22:13:46.150507 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10680,"thread_start_us":10511,"threads_started":1}
I20260927 22:13:46.151108 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:46.131428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43395 (local address 127.0.0.1:45876)
0927 22:13:46.142206 (+ 10778us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:46.142228 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:46.142343 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:46.143602 (+ 1259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:46.143606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:46.143628 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:46.143982 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:46.143991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.145774 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.145780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:46.147995 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:46.148005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.149643 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.149650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:46.149891 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:46.149924 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:13:46.150008 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 22:13:46.150129 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10652,"mutex_wait_us":122,"thread_start_us":6967,"threads_started":1}
I20260927 22:13:46.155007 19292 tablet_service.cc:1467] Processing CreateTablet for tablet c99b9121103647a6bb57b9472527cb6b (DEFAULT_TABLE table=test-table [id=06de564e38834f70bb91aa20547cccfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:13:46.155695 19553 tablet_service.cc:1467] Processing CreateTablet for tablet c99b9121103647a6bb57b9472527cb6b (DEFAULT_TABLE table=test-table [id=06de564e38834f70bb91aa20547cccfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:13:46.155658 19423 tablet_service.cc:1467] Processing CreateTablet for tablet c99b9121103647a6bb57b9472527cb6b (DEFAULT_TABLE table=test-table [id=06de564e38834f70bb91aa20547cccfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:13:46.156706 19292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99b9121103647a6bb57b9472527cb6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:46.156921 19553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99b9121103647a6bb57b9472527cb6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:46.157097 19423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99b9121103647a6bb57b9472527cb6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:46.169691 19636 tablet_bootstrap.cc:492] T c99b9121103647a6bb57b9472527cb6b P 493f9c4e99ce43dcac9823ba9fa28f7e: Bootstrap starting.
I20260927 22:13:46.171562 19637 tablet_bootstrap.cc:492] T c99b9121103647a6bb57b9472527cb6b P 1cfd364dfdb6468a87a4e0b1507e250b: Bootstrap starting.
I20260927 22:13:46.173202 19638 tablet_bootstrap.cc:492] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f: Bootstrap starting.
I20260927 22:13:46.173885 19636 tablet_bootstrap.cc:654] T c99b9121103647a6bb57b9472527cb6b P 493f9c4e99ce43dcac9823ba9fa28f7e: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:46.175161 19637 tablet_bootstrap.cc:654] T c99b9121103647a6bb57b9472527cb6b P 1cfd364dfdb6468a87a4e0b1507e250b: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:46.175218 19636 log.cc:824] T c99b9121103647a6bb57b9472527cb6b P 493f9c4e99ce43dcac9823ba9fa28f7e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:46.176175 19638 tablet_bootstrap.cc:654] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:46.176373 19637 log.cc:824] T c99b9121103647a6bb57b9472527cb6b P 1cfd364dfdb6468a87a4e0b1507e250b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:46.177381 19638 log.cc:824] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:46.178516 19636 tablet_bootstrap.cc:492] T c99b9121103647a6bb57b9472527cb6b P 493f9c4e99ce43dcac9823ba9fa28f7e: No bootstrap required, opened a new log
I20260927 22:13:46.178859 19636 ts_tablet_manager.cc:1397] T c99b9121103647a6bb57b9472527cb6b P 493f9c4e99ce43dcac9823ba9fa28f7e: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260927 22:13:46.179346 19637 tablet_bootstrap.cc:492] T c99b9121103647a6bb57b9472527cb6b P 1cfd364dfdb6468a87a4e0b1507e250b: No bootstrap required, opened a new log
I20260927 22:13:46.179682 19637 ts_tablet_manager.cc:1397] T c99b9121103647a6bb57b9472527cb6b P 1cfd364dfdb6468a87a4e0b1507e250b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20260927 22:13:46.180145 19638 tablet_bootstrap.cc:492] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f: No bootstrap required, opened a new log
I20260927 22:13:46.180449 19638 ts_tablet_manager.cc:1397] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.007s
I20260927 22:13:46.188014 19638 raft_consensus.cc:348] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36799e8cea1a443491b1d6f0d0775f8f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43271 } } peers { permanent_uuid: "1cfd364dfdb6468a87a4e0b1507e250b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38599 } } peers { permanent_uuid: "493f9c4e99ce43dcac9823ba9fa28f7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43395 } }
I20260927 22:13:46.188441 19638 raft_consensus.cc:374] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:46.188575 19638 raft_consensus.cc:729] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36799e8cea1a443491b1d6f0d0775f8f, State: Initialized, Role: FOLLOWER
I20260927 22:13:46.189194 19638 consensus_queue.cc:260] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36799e8cea1a443491b1d6f0d0775f8f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43271 } } peers { permanent_uuid: "1cfd364dfdb6468a87a4e0b1507e250b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38599 } } peers { permanent_uuid: "493f9c4e99ce43dcac9823ba9fa28f7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43395 } }
I20260927 22:13:46.190292 19488 heartbeater.cc:503] Master 127.0.97.62:43637 was elected leader, sending a full tablet report...
I20260927 22:13:46.190510 19636 raft_consensus.cc:348] T c99b9121103647a6bb57b9472527cb6b P 493f9c4e99ce43dcac9823ba9fa28f7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36799e8cea1a443491b1d6f0d0775f8f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43271 } } peers { permanent_uuid: "1cfd364dfdb6468a87a4e0b1507e250b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38599 } } peers { permanent_uuid: "493f9c4e99ce43dcac9823ba9fa28f7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43395 } }
I20260927 22:13:46.190505 19637 raft_consensus.cc:348] T c99b9121103647a6bb57b9472527cb6b P 1cfd364dfdb6468a87a4e0b1507e250b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36799e8cea1a443491b1d6f0d0775f8f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43271 } } peers { permanent_uuid: "1cfd364dfdb6468a87a4e0b1507e250b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38599 } } peers { permanent_uuid: "493f9c4e99ce43dcac9823ba9fa28f7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43395 } }
I20260927 22:13:46.190956 19637 raft_consensus.cc:374] T c99b9121103647a6bb57b9472527cb6b P 1cfd364dfdb6468a87a4e0b1507e250b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:46.190956 19636 raft_consensus.cc:374] T c99b9121103647a6bb57b9472527cb6b P 493f9c4e99ce43dcac9823ba9fa28f7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:46.191056 19636 raft_consensus.cc:729] T c99b9121103647a6bb57b9472527cb6b P 493f9c4e99ce43dcac9823ba9fa28f7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493f9c4e99ce43dcac9823ba9fa28f7e, State: Initialized, Role: FOLLOWER
I20260927 22:13:46.191057 19637 raft_consensus.cc:729] T c99b9121103647a6bb57b9472527cb6b P 1cfd364dfdb6468a87a4e0b1507e250b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfd364dfdb6468a87a4e0b1507e250b, State: Initialized, Role: FOLLOWER
I20260927 22:13:46.191641 19637 consensus_queue.cc:260] T c99b9121103647a6bb57b9472527cb6b P 1cfd364dfdb6468a87a4e0b1507e250b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36799e8cea1a443491b1d6f0d0775f8f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43271 } } peers { permanent_uuid: "1cfd364dfdb6468a87a4e0b1507e250b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38599 } } peers { permanent_uuid: "493f9c4e99ce43dcac9823ba9fa28f7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43395 } }
I20260927 22:13:46.191694 19636 consensus_queue.cc:260] T c99b9121103647a6bb57b9472527cb6b P 493f9c4e99ce43dcac9823ba9fa28f7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36799e8cea1a443491b1d6f0d0775f8f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43271 } } peers { permanent_uuid: "1cfd364dfdb6468a87a4e0b1507e250b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38599 } } peers { permanent_uuid: "493f9c4e99ce43dcac9823ba9fa28f7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43395 } }
I20260927 22:13:46.192466 19638 ts_tablet_manager.cc:1428] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f: Time spent starting tablet: real 0.012s user 0.004s sys 0.006s
I20260927 22:13:46.196635 19637 ts_tablet_manager.cc:1428] T c99b9121103647a6bb57b9472527cb6b P 1cfd364dfdb6468a87a4e0b1507e250b: Time spent starting tablet: real 0.017s user 0.010s sys 0.006s
I20260927 22:13:46.198848 19618 heartbeater.cc:503] Master 127.0.97.62:43637 was elected leader, sending a full tablet report...
I20260927 22:13:46.199533 19636 ts_tablet_manager.cc:1428] T c99b9121103647a6bb57b9472527cb6b P 493f9c4e99ce43dcac9823ba9fa28f7e: Time spent starting tablet: real 0.021s user 0.012s sys 0.004s
I20260927 22:13:46.251230 19642 raft_consensus.cc:482] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:13:46.251619 19642 raft_consensus.cc:504] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36799e8cea1a443491b1d6f0d0775f8f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43271 } } peers { permanent_uuid: "1cfd364dfdb6468a87a4e0b1507e250b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38599 } } peers { permanent_uuid: "493f9c4e99ce43dcac9823ba9fa28f7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43395 } }
I20260927 22:13:46.253643 19642 leader_election.cc:288] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cfd364dfdb6468a87a4e0b1507e250b (127.0.97.1:38599), 493f9c4e99ce43dcac9823ba9fa28f7e (127.0.97.3:43395)
I20260927 22:13:46.262862 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:46.255028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33045 (local address 127.0.97.3:43395)
0927 22:13:46.255241 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:46.255249 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:46.255267 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:13:46.255402 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:46.255408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:46.255489 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:46.255579 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:46.256294 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.257335 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.260441 (+ 3106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.261697 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.261847 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:46.261949 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:46.262448 (+ 499us) server_negotiation.cc:299] Negotiation successful
0927 22:13:46.262584 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:13:46.262916 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:46.254357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43051 (local address 127.0.97.1:38599)
0927 22:13:46.254760 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:46.254769 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:46.255007 (+ 238us) server_negotiation.cc:407] Connection header received
0927 22:13:46.255304 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:46.255309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:46.255406 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:46.255506 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:46.256169 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.257255 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.260439 (+ 3184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.261738 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.261860 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:46.261949 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:46.262448 (+ 499us) server_negotiation.cc:299] Negotiation successful
0927 22:13:46.262584 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260927 22:13:46.263020 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:46.254084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43395 (local address 127.0.97.2:33045)
0927 22:13:46.255145 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:46.255160 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:46.255265 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:46.255606 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:46.255612 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:46.255639 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:46.256126 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:46.256139 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.257491 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.257501 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:46.260184 (+ 2683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:46.260200 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.261985 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.261992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:46.262248 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:46.262301 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 22:13:46.262393 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 22:13:46.262608 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"mutex_wait_us":246,"thread_start_us":759,"threads_started":1}
I20260927 22:13:46.263284 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:46.254000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38599 (local address 127.0.97.2:43051)
0927 22:13:46.254883 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:46.254908 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:46.255125 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:46.255526 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:46.255529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:46.255574 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:46.256020 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:46.256037 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.257439 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.257445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:46.260196 (+ 2751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:46.260204 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.261952 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.261962 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:46.262239 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:46.262295 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 22:13:46.262393 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 22:13:46.262587 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":261,"thread_start_us":468,"threads_started":1}
I20260927 22:13:46.264103 19573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99b9121103647a6bb57b9472527cb6b" candidate_uuid: "36799e8cea1a443491b1d6f0d0775f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493f9c4e99ce43dcac9823ba9fa28f7e" is_pre_election: true
I20260927 22:13:46.264358 19312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99b9121103647a6bb57b9472527cb6b" candidate_uuid: "36799e8cea1a443491b1d6f0d0775f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfd364dfdb6468a87a4e0b1507e250b" is_pre_election: true
I20260927 22:13:46.264628 19573 raft_consensus.cc:2454] T c99b9121103647a6bb57b9472527cb6b P 493f9c4e99ce43dcac9823ba9fa28f7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36799e8cea1a443491b1d6f0d0775f8f in term 0.
I20260927 22:13:46.264955 19312 raft_consensus.cc:2454] T c99b9121103647a6bb57b9472527cb6b P 1cfd364dfdb6468a87a4e0b1507e250b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36799e8cea1a443491b1d6f0d0775f8f in term 0.
I20260927 22:13:46.265415 19378 leader_election.cc:302] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36799e8cea1a443491b1d6f0d0775f8f, 493f9c4e99ce43dcac9823ba9fa28f7e; no voters:
I20260927 22:13:46.265909 19642 raft_consensus.cc:2781] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:13:46.266028 19642 raft_consensus.cc:482] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:13:46.266146 19642 raft_consensus.cc:3037] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:46.268450 19642 raft_consensus.cc:504] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36799e8cea1a443491b1d6f0d0775f8f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43271 } } peers { permanent_uuid: "1cfd364dfdb6468a87a4e0b1507e250b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38599 } } peers { permanent_uuid: "493f9c4e99ce43dcac9823ba9fa28f7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43395 } }
I20260927 22:13:46.269150 19642 leader_election.cc:288] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f [CANDIDATE]: Term 1 election: Requested vote from peers 1cfd364dfdb6468a87a4e0b1507e250b (127.0.97.1:38599), 493f9c4e99ce43dcac9823ba9fa28f7e (127.0.97.3:43395)
I20260927 22:13:46.269604 19312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99b9121103647a6bb57b9472527cb6b" candidate_uuid: "36799e8cea1a443491b1d6f0d0775f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfd364dfdb6468a87a4e0b1507e250b"
I20260927 22:13:46.269706 19573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99b9121103647a6bb57b9472527cb6b" candidate_uuid: "36799e8cea1a443491b1d6f0d0775f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493f9c4e99ce43dcac9823ba9fa28f7e"
I20260927 22:13:46.269810 19312 raft_consensus.cc:3037] T c99b9121103647a6bb57b9472527cb6b P 1cfd364dfdb6468a87a4e0b1507e250b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:46.269984 19573 raft_consensus.cc:3037] T c99b9121103647a6bb57b9472527cb6b P 493f9c4e99ce43dcac9823ba9fa28f7e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:46.272380 19312 raft_consensus.cc:2454] T c99b9121103647a6bb57b9472527cb6b P 1cfd364dfdb6468a87a4e0b1507e250b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36799e8cea1a443491b1d6f0d0775f8f in term 1.
I20260927 22:13:46.272464 19573 raft_consensus.cc:2454] T c99b9121103647a6bb57b9472527cb6b P 493f9c4e99ce43dcac9823ba9fa28f7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36799e8cea1a443491b1d6f0d0775f8f in term 1.
I20260927 22:13:46.272832 19376 leader_election.cc:302] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfd364dfdb6468a87a4e0b1507e250b, 36799e8cea1a443491b1d6f0d0775f8f; no voters:
I20260927 22:13:46.273207 19642 raft_consensus.cc:2781] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:13:46.273696 19642 raft_consensus.cc:686] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f [term 1 LEADER]: Becoming Leader. State: Replica: 36799e8cea1a443491b1d6f0d0775f8f, State: Running, Role: LEADER
I20260927 22:13:46.274288 19642 consensus_queue.cc:237] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36799e8cea1a443491b1d6f0d0775f8f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43271 } } peers { permanent_uuid: "1cfd364dfdb6468a87a4e0b1507e250b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38599 } } peers { permanent_uuid: "493f9c4e99ce43dcac9823ba9fa28f7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43395 } }
I20260927 22:13:46.278892 19171 catalog_manager.cc:5495] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 36799e8cea1a443491b1d6f0d0775f8f (127.0.97.2). New cstate: current_term: 1 leader_uuid: "36799e8cea1a443491b1d6f0d0775f8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36799e8cea1a443491b1d6f0d0775f8f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cfd364dfdb6468a87a4e0b1507e250b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "493f9c4e99ce43dcac9823ba9fa28f7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43395 } health_report { overall_health: UNKNOWN } } }
I20260927 22:13:46.295425 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:46.289762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36340 (local address 127.0.97.2:43271)
0927 22:13:46.289959 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:46.289966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:46.290064 (+ 98us) server_negotiation.cc:407] Connection header received
0927 22:13:46.290299 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:46.290303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:46.290420 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:46.290580 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:13:46.290984 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.291693 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.293098 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.293527 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.293661 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:46.294235 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:13:46.294827 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:13:46.295012 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:46.295127 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 22:13:46.295173 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:13:46.304229 19312 raft_consensus.cc:1261] T c99b9121103647a6bb57b9472527cb6b P 1cfd364dfdb6468a87a4e0b1507e250b [term 1 FOLLOWER]: Refusing update from remote peer 36799e8cea1a443491b1d6f0d0775f8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:13:46.304230 19573 raft_consensus.cc:1261] T c99b9121103647a6bb57b9472527cb6b P 493f9c4e99ce43dcac9823ba9fa28f7e [term 1 FOLLOWER]: Refusing update from remote peer 36799e8cea1a443491b1d6f0d0775f8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:13:46.305377 19647 consensus_queue.cc:1035] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "493f9c4e99ce43dcac9823ba9fa28f7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:13:46.305706 19642 consensus_queue.cc:1035] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cfd364dfdb6468a87a4e0b1507e250b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:13:46.316502 19619 tablet.cc:2369] T c99b9121103647a6bb57b9472527cb6b P 493f9c4e99ce43dcac9823ba9fa28f7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:13:46.317973 19650 mvcc.cc:205] Tried to move back new op lower bound from 7334081438930853888 to 7334081438823571456. Current Snapshot: MvccSnapshot[applied={T|T < 7334081438930853888}]
I20260927 22:13:46.318789 19651 mvcc.cc:205] Tried to move back new op lower bound from 7334081438930853888 to 7334081438823571456. Current Snapshot: MvccSnapshot[applied={T|T < 7334081438930853888}]
I20260927 22:13:46.328379 19652 mvcc.cc:205] Tried to move back new op lower bound from 7334081438930853888 to 7334081438823571456. Current Snapshot: MvccSnapshot[applied={T|T < 7334081438930853888}]
I20260927 22:13:46.341158 19171 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:13:46.341450 19171 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:13:46.345026 19171 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4be380cd3add4229857dab88980174dc: Sending DeleteTablet for 3 replicas of tablet c99b9121103647a6bb57b9472527cb6b
I20260927 22:13:46.346302 19423 tablet_service.cc:1514] Processing DeleteTablet for tablet c99b9121103647a6bb57b9472527cb6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:13:46 UTC) from {username='slave'} at 127.0.0.1:36336
I20260927 22:13:46.346261 19292 tablet_service.cc:1514] Processing DeleteTablet for tablet c99b9121103647a6bb57b9472527cb6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:13:46 UTC) from {username='slave'} at 127.0.0.1:41614
I20260927 22:13:46.347604 19553 tablet_service.cc:1514] Processing DeleteTablet for tablet c99b9121103647a6bb57b9472527cb6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:13:46 UTC) from {username='slave'} at 127.0.0.1:45876
I20260927 22:13:46.347982 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 19230
I20260927 22:13:46.357399 19662 tablet_replica.cc:330] T c99b9121103647a6bb57b9472527cb6b P 493f9c4e99ce43dcac9823ba9fa28f7e: stopping tablet replica
I20260927 22:13:46.357966 19662 raft_consensus.cc:2229] T c99b9121103647a6bb57b9472527cb6b P 493f9c4e99ce43dcac9823ba9fa28f7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:13:46.360975 19661 tablet_replica.cc:330] T c99b9121103647a6bb57b9472527cb6b P 1cfd364dfdb6468a87a4e0b1507e250b: stopping tablet replica
I20260927 22:13:46.361538 19661 raft_consensus.cc:2229] T c99b9121103647a6bb57b9472527cb6b P 1cfd364dfdb6468a87a4e0b1507e250b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:13:46.362327 19660 tablet_replica.cc:330] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f: stopping tablet replica
I20260927 22:13:46.362550 19662 raft_consensus.cc:2258] T c99b9121103647a6bb57b9472527cb6b P 493f9c4e99ce43dcac9823ba9fa28f7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:13:46.363845 19660 raft_consensus.cc:2229] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:13:46.364601 19660 raft_consensus.cc:2258] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:13:46.365168 19661 raft_consensus.cc:2258] T c99b9121103647a6bb57b9472527cb6b P 1cfd364dfdb6468a87a4e0b1507e250b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:13:46.365310 19662 ts_tablet_manager.cc:1905] T c99b9121103647a6bb57b9472527cb6b P 493f9c4e99ce43dcac9823ba9fa28f7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:13:46.367587 19660 ts_tablet_manager.cc:1905] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:13:46.370589 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:46.349696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41628 (local address 127.0.97.1:38599)
0927 22:13:46.349930 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:46.349940 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:46.361700 (+ 11760us) server_negotiation.cc:407] Connection header received
0927 22:13:46.362005 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:46.362012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:46.362146 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:46.362230 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:46.362938 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.364186 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.366220 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.366933 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.367034 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:46.368786 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:46.368799 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:46.368819 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:46.369636 (+ 817us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:46.369697 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:46.369724 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:46.369813 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:46.370065 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 22:13:46.370188 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:13:46.371548 19661 ts_tablet_manager.cc:1905] T c99b9121103647a6bb57b9472527cb6b P 1cfd364dfdb6468a87a4e0b1507e250b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:13:46.372023 19352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:46.373924 19662 ts_tablet_manager.cc:1918] T c99b9121103647a6bb57b9472527cb6b P 493f9c4e99ce43dcac9823ba9fa28f7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:13:46.374156 19662 log.cc:1196] T c99b9121103647a6bb57b9472527cb6b P 493f9c4e99ce43dcac9823ba9fa28f7e: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/c99b9121103647a6bb57b9472527cb6b
I20260927 22:13:46.374825 19662 ts_tablet_manager.cc:1939] T c99b9121103647a6bb57b9472527cb6b P 493f9c4e99ce43dcac9823ba9fa28f7e: Deleting consensus metadata
I20260927 22:13:46.376998 19157 catalog_manager.cc:4841] TS 493f9c4e99ce43dcac9823ba9fa28f7e (127.0.97.3:43395): tablet c99b9121103647a6bb57b9472527cb6b (table test-table [id=06de564e38834f70bb91aa20547cccfe]) successfully deleted
I20260927 22:13:46.380396 19660 ts_tablet_manager.cc:1918] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:13:46.380604 19660 log.cc:1196] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/c99b9121103647a6bb57b9472527cb6b
I20260927 22:13:46.381274 19660 ts_tablet_manager.cc:1939] T c99b9121103647a6bb57b9472527cb6b P 36799e8cea1a443491b1d6f0d0775f8f: Deleting consensus metadata
I20260927 22:13:46.383088 19156 catalog_manager.cc:4841] TS 36799e8cea1a443491b1d6f0d0775f8f (127.0.97.2:43271): tablet c99b9121103647a6bb57b9472527cb6b (table test-table [id=06de564e38834f70bb91aa20547cccfe]) successfully deleted
I20260927 22:13:46.504048 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 19230
W20260927 22:13:46.522400 19155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:38599 (error 108)
I20260927 22:13:46.522675 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 19361
I20260927 22:13:46.524857 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:46.524255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38599 (local address 127.0.0.1:41632)
0927 22:13:46.524538 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:46.524657 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
W20260927 22:13:46.525149 19155 catalog_manager.cc:4584] TS 1cfd364dfdb6468a87a4e0b1507e250b (127.0.97.1:38599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c99b9121103647a6bb57b9472527cb6b: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38599: connect: Connection refused (error 111)
I20260927 22:13:46.536609 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:46.523768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36344 (local address 127.0.97.2:43271)
0927 22:13:46.523956 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:46.523963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:46.523990 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:13:46.524061 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:46.524067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:46.524157 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:46.524260 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:46.524758 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.525788 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.527380 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.527867 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.528014 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:46.535123 (+ 7109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:46.535133 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:46.535149 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:46.535769 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:46.535819 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:46.535839 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:46.536008 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:46.536155 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 22:13:46.536265 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:13:46.537397 19483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:46.579047 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:46.578512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38599 (local address 127.0.0.1:41646)
0927 22:13:46.578766 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:46.578883 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260927 22:13:46.580300 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:46.579934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38599 (local address 127.0.0.1:41648)
0927 22:13:46.580097 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:46.580164 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 22:13:46.641343 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:46.640747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38599 (local address 127.0.0.1:41658)
0927 22:13:46.641013 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:46.641136 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 22:13:46.642647 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:46.642229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38599 (local address 127.0.0.1:41674)
0927 22:13:46.642419 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:46.642494 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 22:13:46.682928 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 19361
I20260927 22:13:46.702351 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 19491
I20260927 22:13:46.709231 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:46.703181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45890 (local address 127.0.97.3:43395)
0927 22:13:46.703374 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:46.703379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:46.703512 (+ 133us) server_negotiation.cc:407] Connection header received
0927 22:13:46.703623 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:46.703628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:46.703710 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:46.703844 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:46.704287 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.705031 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.706248 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:46.706815 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:46.706913 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:46.708024 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:46.708033 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:46.708049 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:46.708623 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:46.708671 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:46.708689 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:46.708775 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:46.708891 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 22:13:46.708980 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:13:46.709895 19613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:46.723331 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:46.722683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38599 (local address 127.0.0.1:41676)
0927 22:13:46.722962 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:46.723099 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 22:13:46.724438 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:46.724012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38599 (local address 127.0.0.1:41680)
0927 22:13:46.724218 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:46.724319 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 22:13:46.799651 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 19491
I20260927 22:13:46.816318 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 19140
I20260927 22:13:46.817422 19201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:46.929628 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:46.929169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38599 (local address 127.0.0.1:41690)
0927 22:13:46.929346 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:46.929442 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 22:13:46.929656 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 19140
2026-09-27T22:13:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2873 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:13:47 dist-test-slave-mp03 krb5kdc[19671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:13:47 dist-test-slave-mp03 krb5kdc[19671](info): set up 2 sockets
Sep 27 22:13:47 dist-test-slave-mp03 krb5kdc[19671](info): commencing operation
krb5kdc: starting...
W20260927 22:13:48.050879 388 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:13:48 dist-test-slave-mp03 krb5kdc[19671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:13:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:13:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:13:48.310914 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36509
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:13:48.652576 19684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:48.652894 19684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:48.652982 19684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:48.660211 19684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:13:48.660320 19684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:48.660372 19684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:48.660422 19684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:13:48.660466 19684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:13:48.670394 19684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:48.671757 19684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:48.673049 19684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:48.684779 19689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:48.684854 19690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:48.684876 19692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:48.686185 19684 server_base.cc:1034] running on GCE node
I20260927 22:13:48.687069 19684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:48.688510 19684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:48.689772 19684 hybrid_clock.cc:648] HybridClock initialized: now 1790547228689731 us; error 36 us; skew 500 ppm
I20260927 22:13:48.690157 19684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:13:48 dist-test-slave-mp03 krb5kdc[19671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:13:48.701993 19684 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:13:48.703788 19684 webserver.cc:460] Webserver started at http://127.0.97.62:33321/ using document root <none> and password file <none>
I20260927 22:13:48.704550 19684 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:48.704698 19684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:48.705019 19684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:48.708137 19684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "17698107cf1b410d91bb5f1bb9b49b2b"
format_stamp: "Formatted at 2026-09-27 22:13:48 on dist-test-slave-mp03"
server_key: "d8c551fcc291b503d16aac74bd3dbd42"
server_key_iv: "0a3d0bb05b941efc508ad7a83b2eb64e"
server_key_version: "encryptionkey@0"
I20260927 22:13:48.708895 19684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "17698107cf1b410d91bb5f1bb9b49b2b"
format_stamp: "Formatted at 2026-09-27 22:13:48 on dist-test-slave-mp03"
server_key: "d8c551fcc291b503d16aac74bd3dbd42"
server_key_iv: "0a3d0bb05b941efc508ad7a83b2eb64e"
server_key_version: "encryptionkey@0"
I20260927 22:13:48.714195 19684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 22:13:48.717823 19699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:48.719470 19684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:13:48.719648 19684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "17698107cf1b410d91bb5f1bb9b49b2b"
format_stamp: "Formatted at 2026-09-27 22:13:48 on dist-test-slave-mp03"
server_key: "d8c551fcc291b503d16aac74bd3dbd42"
server_key_iv: "0a3d0bb05b941efc508ad7a83b2eb64e"
server_key_version: "encryptionkey@0"
I20260927 22:13:48.719885 19684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:48.747208 19684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:48.751971 19684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:48.752271 19684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:48.771685 19684 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36509
I20260927 22:13:48.771716 19750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36509 every 8 connection(s)
I20260927 22:13:48.773185 19684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:13:48.777081 19751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:48.780364 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 19684
I20260927 22:13:48.780560 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:13:48.781056 388 external_mini_cluster.cc:1428] Setting key f2ef7bd6e8bb9f29fb40865e97179768
I20260927 22:13:48.790290 19751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17698107cf1b410d91bb5f1bb9b49b2b: Bootstrap starting.
I20260927 22:13:48.793929 19751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17698107cf1b410d91bb5f1bb9b49b2b: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:48.795033 19751 log.cc:824] T 00000000000000000000000000000000 P 17698107cf1b410d91bb5f1bb9b49b2b: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:13:48 dist-test-slave-mp03 krb5kdc[19671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:13:48.798388 19751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17698107cf1b410d91bb5f1bb9b49b2b: No bootstrap required, opened a new log
I20260927 22:13:48.807262 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:48.783186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51840 (local address 127.0.97.62:36509)
0927 22:13:48.783964 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:48.783979 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:48.784031 (+ 52us) server_negotiation.cc:407] Connection header received
0927 22:13:48.786022 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:48.786052 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:48.786642 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:48.787107 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:48.788473 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:48.790457 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:48.791853 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:48.792926 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:48.793117 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:48.797635 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:48.797647 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:48.797670 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:48.797719 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:48.802107 (+ 4388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:48.803046 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:48.803056 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:48.803066 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:48.803302 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:48.803763 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:48.803767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:48.803770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:48.804553 (+ 783us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:48.804859 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:48.805277 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 22:13:48.805772 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":318,"threads_started":1}
I20260927 22:13:48.809139 19751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17698107cf1b410d91bb5f1bb9b49b2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17698107cf1b410d91bb5f1bb9b49b2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36509 } }
I20260927 22:13:48.809528 19751 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17698107cf1b410d91bb5f1bb9b49b2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:48.809651 19751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17698107cf1b410d91bb5f1bb9b49b2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17698107cf1b410d91bb5f1bb9b49b2b, State: Initialized, Role: FOLLOWER
I20260927 22:13:48.810328 19751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17698107cf1b410d91bb5f1bb9b49b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17698107cf1b410d91bb5f1bb9b49b2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36509 } }
I20260927 22:13:48.810631 19751 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17698107cf1b410d91bb5f1bb9b49b2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:13:48.810763 19751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17698107cf1b410d91bb5f1bb9b49b2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:13:48.810947 19751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17698107cf1b410d91bb5f1bb9b49b2b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:48.813400 19751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17698107cf1b410d91bb5f1bb9b49b2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17698107cf1b410d91bb5f1bb9b49b2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36509 } }
I20260927 22:13:48.813898 19751 leader_election.cc:302] T 00000000000000000000000000000000 P 17698107cf1b410d91bb5f1bb9b49b2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17698107cf1b410d91bb5f1bb9b49b2b; no voters:
I20260927 22:13:48.814502 19751 leader_election.cc:288] T 00000000000000000000000000000000 P 17698107cf1b410d91bb5f1bb9b49b2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:13:48.814685 19756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17698107cf1b410d91bb5f1bb9b49b2b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:13:48.815510 19756 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17698107cf1b410d91bb5f1bb9b49b2b [term 1 LEADER]: Becoming Leader. State: Replica: 17698107cf1b410d91bb5f1bb9b49b2b, State: Running, Role: LEADER
I20260927 22:13:48.816102 19756 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17698107cf1b410d91bb5f1bb9b49b2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17698107cf1b410d91bb5f1bb9b49b2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36509 } }
I20260927 22:13:48.816680 19751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17698107cf1b410d91bb5f1bb9b49b2b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:13:48.819198 19758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17698107cf1b410d91bb5f1bb9b49b2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17698107cf1b410d91bb5f1bb9b49b2b. Latest consensus state: current_term: 1 leader_uuid: "17698107cf1b410d91bb5f1bb9b49b2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17698107cf1b410d91bb5f1bb9b49b2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36509 } } }
I20260927 22:13:48.819737 19758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17698107cf1b410d91bb5f1bb9b49b2b [sys.catalog]: This master's current role is: LEADER
I20260927 22:13:48.819248 19757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17698107cf1b410d91bb5f1bb9b49b2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17698107cf1b410d91bb5f1bb9b49b2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17698107cf1b410d91bb5f1bb9b49b2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36509 } } }
I20260927 22:13:48.820142 19757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17698107cf1b410d91bb5f1bb9b49b2b [sys.catalog]: This master's current role is: LEADER
I20260927 22:13:48.820865 19765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:13:48.829799 19765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:13:48.838850 19765 catalog_manager.cc:1347] Generated new cluster ID: 9ab70296006c42549fc5ea7713851843
I20260927 22:13:48.838989 19765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:13:48.861443 19765 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:13:48.863587 19765 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:13:48.875571 19765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17698107cf1b410d91bb5f1bb9b49b2b: Generated new TSK 0
I20260927 22:13:48.876443 19765 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:13:48.955813 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36509
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:13:49.308383 19777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:49.308662 19777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:49.308753 19777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:49.315886 19777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:49.315995 19777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:49.316107 19777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:13:49.326459 19777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:49.328109 19777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:49.330060 19777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:49.343605 19783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:49.343740 19782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:49.343762 19785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:49.344791 19777 server_base.cc:1034] running on GCE node
I20260927 22:13:49.345412 19777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:49.346347 19777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:49.347571 19777 hybrid_clock.cc:648] HybridClock initialized: now 1790547229347494 us; error 70 us; skew 500 ppm
I20260927 22:13:49.347888 19777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:13:49 dist-test-slave-mp03 krb5kdc[19671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:13:49.351073 19777 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:13:49.352587 19777 webserver.cc:460] Webserver started at http://127.0.97.1:43869/ using document root <none> and password file <none>
I20260927 22:13:49.353394 19777 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:49.353533 19777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:49.353843 19777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:49.356783 19777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "ef25b78f79114e9282107cf292d523c6"
format_stamp: "Formatted at 2026-09-27 22:13:49 on dist-test-slave-mp03"
server_key: "7fd3029cd64f8e5d6881db33bd5eda9e"
server_key_iv: "dbf779af2528ee3ec3cf78bc1fb29258"
server_key_version: "encryptionkey@0"
I20260927 22:13:49.357569 19777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ef25b78f79114e9282107cf292d523c6"
format_stamp: "Formatted at 2026-09-27 22:13:49 on dist-test-slave-mp03"
server_key: "7fd3029cd64f8e5d6881db33bd5eda9e"
server_key_iv: "dbf779af2528ee3ec3cf78bc1fb29258"
server_key_version: "encryptionkey@0"
I20260927 22:13:49.362831 19777 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260927 22:13:49.366402 19792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:49.367887 19777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:13:49.368080 19777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "ef25b78f79114e9282107cf292d523c6"
format_stamp: "Formatted at 2026-09-27 22:13:49 on dist-test-slave-mp03"
server_key: "7fd3029cd64f8e5d6881db33bd5eda9e"
server_key_iv: "dbf779af2528ee3ec3cf78bc1fb29258"
server_key_version: "encryptionkey@0"
I20260927 22:13:49.368299 19777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:49.387765 19777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:49.392379 19777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:49.392714 19777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:49.393718 19777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:13:49.395151 19777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:13:49.395224 19777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:49.395391 19777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:13:49.395452 19777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:49.427865 19777 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38019
I20260927 22:13:49.428014 19904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38019 every 8 connection(s)
I20260927 22:13:49.429203 19777 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:13:49.435016 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 19777
I20260927 22:13:49.435190 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:13:49.435640 388 external_mini_cluster.cc:1428] Setting key 55f928b6fc65a47742abf1199774f0b4
Sep 27 22:13:49 dist-test-slave-mp03 krb5kdc[19671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:13:49.451691 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:49.432027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48981 (local address 127.0.97.62:36509)
0927 22:13:49.432527 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:49.432534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:49.433502 (+ 968us) server_negotiation.cc:407] Connection header received
0927 22:13:49.435609 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:49.435615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:49.435715 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:49.435882 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:49.438023 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:49.438908 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:49.440596 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:49.441040 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:49.441201 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:49.445410 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:49.445415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:49.445419 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:49.445453 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:49.448108 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:49.449170 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:49.449174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:49.449176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:49.449314 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:49.449866 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:49.449869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:49.449871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:49.450130 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:49.450243 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:49.451231 (+ 988us) server_negotiation.cc:299] Negotiation successful
0927 22:13:49.451415 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":235,"threads_started":1}
I20260927 22:13:49.453393 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:49.432614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36509 (local address 127.0.97.1:48981)
0927 22:13:49.433338 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:49.433383 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:49.435334 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:49.436123 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:49.436131 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:49.436846 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:13:49.437796 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:49.437831 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:49.439090 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:49.439097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:49.440432 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:49.440446 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:49.441311 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:49.441322 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:49.441600 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:49.442379 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:13:49.442409 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:13:49.445115 (+ 2706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:13:49.448256 (+ 3141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:13:49.448269 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:13:49.448288 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:13:49.449030 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:13:49.449450 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:13:49.449458 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:13:49.449464 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:13:49.449735 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:13:49.450264 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:13:49.450276 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:13:49.450803 (+ 527us) client_negotiation.cc:769] Sending connection context
0927 22:13:49.451156 (+ 353us) client_negotiation.cc:240] Negotiation successful
0927 22:13:49.451799 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":277,"threads_started":1}
I20260927 22:13:49.455235 19905 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36509
I20260927 22:13:49.455605 19905 heartbeater.cc:464] Registering TS with master...
I20260927 22:13:49.456408 19905 heartbeater.cc:511] Master 127.0.97.62:36509 requested a full tablet report, sending...
I20260927 22:13:49.458645 19716 ts_manager.cc:194] Registered new tserver with Master: ef25b78f79114e9282107cf292d523c6 (127.0.97.1:38019)
I20260927 22:13:49.460151 19716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48981
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:13:49.497714 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36509
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:13:49.828738 19911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:49.829070 19911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:49.829167 19911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:49.836550 19911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:49.836676 19911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:49.836790 19911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:13:49.846658 19911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:49.847919 19911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:49.849179 19911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:49.862274 19916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:49.862305 19919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:49.862274 19917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:49.863780 19911 server_base.cc:1034] running on GCE node
I20260927 22:13:49.864477 19911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:49.865512 19911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:49.866719 19911 hybrid_clock.cc:648] HybridClock initialized: now 1790547229866661 us; error 52 us; skew 500 ppm
I20260927 22:13:49.867084 19911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:13:49 dist-test-slave-mp03 krb5kdc[19671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:13:49.870820 19911 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:13:49.872582 19911 webserver.cc:460] Webserver started at http://127.0.97.2:41423/ using document root <none> and password file <none>
I20260927 22:13:49.873369 19911 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:49.873529 19911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:49.873868 19911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:49.877164 19911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "d9c7b595aade4eca8a2d2268c0a8194a"
format_stamp: "Formatted at 2026-09-27 22:13:49 on dist-test-slave-mp03"
server_key: "ce5f3c2180d860266b70db9e1e92b23a"
server_key_iv: "4fe6c1a1c7b7a5330ec532910d29b544"
server_key_version: "encryptionkey@0"
I20260927 22:13:49.878054 19911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d9c7b595aade4eca8a2d2268c0a8194a"
format_stamp: "Formatted at 2026-09-27 22:13:49 on dist-test-slave-mp03"
server_key: "ce5f3c2180d860266b70db9e1e92b23a"
server_key_iv: "4fe6c1a1c7b7a5330ec532910d29b544"
server_key_version: "encryptionkey@0"
I20260927 22:13:49.883947 19911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 22:13:49.887730 19926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:49.889065 19911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:13:49.889261 19911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "d9c7b595aade4eca8a2d2268c0a8194a"
format_stamp: "Formatted at 2026-09-27 22:13:49 on dist-test-slave-mp03"
server_key: "ce5f3c2180d860266b70db9e1e92b23a"
server_key_iv: "4fe6c1a1c7b7a5330ec532910d29b544"
server_key_version: "encryptionkey@0"
I20260927 22:13:49.889467 19911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:49.901952 19911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:49.906502 19911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:49.906847 19911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:49.907828 19911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:13:49.909201 19911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:13:49.909304 19911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:49.909458 19911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:13:49.909530 19911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:49.943351 19911 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44755
I20260927 22:13:49.943434 20038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44755 every 8 connection(s)
I20260927 22:13:49.944614 19911 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:13:49.952860 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 19911
I20260927 22:13:49.953048 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:13:49.953539 388 external_mini_cluster.cc:1428] Setting key e475160baaf24a0c415af1b434b89810
Sep 27 22:13:49 dist-test-slave-mp03 krb5kdc[19671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:13:49.967170 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:49.947159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39983 (local address 127.0.97.62:36509)
0927 22:13:49.947341 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:49.947346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:49.948471 (+ 1125us) server_negotiation.cc:407] Connection header received
0927 22:13:49.950329 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:49.950334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:49.950438 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:49.950551 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:49.952026 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:49.953010 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:49.954894 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:49.955564 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:49.955672 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:49.960018 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:49.960027 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:49.960035 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:49.960083 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:49.963716 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:49.964588 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:49.964596 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:49.964603 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:49.964760 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:49.965327 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:49.965330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:49.965332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:49.965622 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:49.965729 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:49.966730 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0927 22:13:49.966910 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:13:49.968734 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:49.947657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36509 (local address 127.0.97.2:39983)
0927 22:13:49.948307 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:49.948345 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:49.950063 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:49.950807 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:49.950815 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:49.951301 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:13:49.951843 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:49.951860 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:49.953192 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:49.953196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:49.954755 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:49.954768 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:49.955813 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:49.955820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:49.956104 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:49.956887 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:13:49.956921 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:13:49.959765 (+ 2844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:13:49.963852 (+ 4087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:13:49.963859 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:13:49.963918 (+ 59us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:13:49.964451 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:13:49.964931 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:13:49.964939 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:13:49.964944 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:13:49.965209 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:13:49.965747 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:13:49.965756 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:13:49.966228 (+ 472us) client_negotiation.cc:769] Sending connection context
0927 22:13:49.966564 (+ 336us) client_negotiation.cc:240] Negotiation successful
0927 22:13:49.967198 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":252,"threads_started":1}
I20260927 22:13:49.970297 20039 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36509
I20260927 22:13:49.970672 20039 heartbeater.cc:464] Registering TS with master...
I20260927 22:13:49.971418 20039 heartbeater.cc:511] Master 127.0.97.62:36509 requested a full tablet report, sending...
I20260927 22:13:49.973045 19716 ts_manager.cc:194] Registered new tserver with Master: d9c7b595aade4eca8a2d2268c0a8194a (127.0.97.2:44755)
I20260927 22:13:49.973927 19716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39983
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:13:50.014837 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36509
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:13:50.343099 20044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:50.343382 20044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:50.343577 20044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:50.350823 20044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:50.350957 20044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:50.351153 20044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:13:50.360991 20044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:50.362332 20044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:50.363683 20044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:50.376854 20049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:50.376814 20052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:50.376920 20050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:50.377947 20044 server_base.cc:1034] running on GCE node
I20260927 22:13:50.378561 20044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:50.379560 20044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:50.380800 20044 hybrid_clock.cc:648] HybridClock initialized: now 1790547230380731 us; error 52 us; skew 500 ppm
I20260927 22:13:50.381129 20044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:13:50 dist-test-slave-mp03 krb5kdc[19671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:13:50.384248 20044 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:13:50.385628 20044 webserver.cc:460] Webserver started at http://127.0.97.3:38387/ using document root <none> and password file <none>
I20260927 22:13:50.386319 20044 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:50.386466 20044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:50.386857 20044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:50.389773 20044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "fb96f7a76df34474aefd99bbc14977ab"
format_stamp: "Formatted at 2026-09-27 22:13:50 on dist-test-slave-mp03"
server_key: "fa6929666be2931fb5c6cc00d10b74fc"
server_key_iv: "53053e3d9e397f795bdef1874c9763ee"
server_key_version: "encryptionkey@0"
I20260927 22:13:50.390494 20044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fb96f7a76df34474aefd99bbc14977ab"
format_stamp: "Formatted at 2026-09-27 22:13:50 on dist-test-slave-mp03"
server_key: "fa6929666be2931fb5c6cc00d10b74fc"
server_key_iv: "53053e3d9e397f795bdef1874c9763ee"
server_key_version: "encryptionkey@0"
I20260927 22:13:50.396042 20044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:13:50.399691 20059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:50.401226 20044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:13:50.401437 20044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "fb96f7a76df34474aefd99bbc14977ab"
format_stamp: "Formatted at 2026-09-27 22:13:50 on dist-test-slave-mp03"
server_key: "fa6929666be2931fb5c6cc00d10b74fc"
server_key_iv: "53053e3d9e397f795bdef1874c9763ee"
server_key_version: "encryptionkey@0"
I20260927 22:13:50.401680 20044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:50.415570 20044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:50.420482 20044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:50.420827 20044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:50.421833 20044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:13:50.423452 20044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:13:50.423593 20044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:50.423756 20044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:13:50.423841 20044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:50.459357 20044 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33955
I20260927 22:13:50.459419 20171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33955 every 8 connection(s)
I20260927 22:13:50.460685 20044 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:13:50.463327 19905 heartbeater.cc:503] Master 127.0.97.62:36509 was elected leader, sending a full tablet report...
I20260927 22:13:50.469149 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 20044
I20260927 22:13:50.469334 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:13:50.469913 388 external_mini_cluster.cc:1428] Setting key d043034c41c8b9359fece62afb215ed6
Sep 27 22:13:50 dist-test-slave-mp03 krb5kdc[19671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:13:50.480904 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:50.463230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41673 (local address 127.0.97.62:36509)
0927 22:13:50.463469 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:50.463476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:50.464777 (+ 1301us) server_negotiation.cc:407] Connection header received
0927 22:13:50.466710 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:50.466714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:50.466774 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:50.466846 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:50.468270 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.469043 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.471013 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.471577 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.471717 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:50.475183 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:50.475188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:50.475191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:50.475222 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:50.477823 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:50.478547 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:50.478556 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:50.478559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:50.478732 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:50.479160 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:50.479164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:50.479168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:50.479440 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:50.479618 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:50.480310 (+ 692us) server_negotiation.cc:299] Negotiation successful
0927 22:13:50.480543 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:13:50.481863 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:50.463732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36509 (local address 127.0.97.3:41673)
0927 22:13:50.464603 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:50.464652 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:50.466408 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:50.467068 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:50.467075 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:50.467545 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:13:50.468096 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:50.468113 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.469208 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.469212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:50.470877 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:50.470890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.471806 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.471815 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:50.472074 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:50.472782 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:13:50.472816 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:13:50.474982 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:13:50.477978 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:13:50.477987 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:13:50.478002 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:13:50.478420 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:13:50.478842 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:13:50.478848 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:13:50.478852 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:13:50.479039 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:13:50.479575 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:13:50.479583 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:13:50.479926 (+ 343us) client_negotiation.cc:769] Sending connection context
0927 22:13:50.480193 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 22:13:50.480754 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":350,"threads_started":1}
I20260927 22:13:50.483453 20172 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36509
I20260927 22:13:50.483798 20172 heartbeater.cc:464] Registering TS with master...
I20260927 22:13:50.484679 20172 heartbeater.cc:511] Master 127.0.97.62:36509 requested a full tablet report, sending...
I20260927 22:13:50.486105 19716 ts_manager.cc:194] Registered new tserver with Master: fb96f7a76df34474aefd99bbc14977ab (127.0.97.3:33955)
I20260927 22:13:50.487001 19716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41673
I20260927 22:13:50.497468 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:13:50.514351 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:50.501538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51850 (local address 127.0.97.62:36509)
0927 22:13:50.501752 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:50.501760 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:50.502141 (+ 381us) server_negotiation.cc:407] Connection header received
0927 22:13:50.502344 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:50.502349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:50.502425 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:50.502528 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:50.503933 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.504634 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.506042 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.506495 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.506683 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:50.509166 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:50.509170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:50.509173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:50.509205 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:50.511831 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:50.512446 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:50.512450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:50.512474 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:50.512589 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:50.512902 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:50.512905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:50.512907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:50.513199 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:50.513400 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:50.513661 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 22:13:50.513914 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:13:50.518204 19716 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:13:50.525799 19716 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:13:50.560056 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:50.538195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33955 (local address 127.0.0.1:48228)
0927 22:13:50.551625 (+ 13430us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:50.551642 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:50.551787 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:50.552713 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:50.552719 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:50.552744 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:50.553315 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:50.553324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.554900 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.554906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:50.557098 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:50.557112 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.559003 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.559010 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:50.559189 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:50.559251 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 22:13:50.559377 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 22:13:50.559571 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13284,"mutex_wait_us":103,"thread_start_us":11378,"threads_started":1}
I20260927 22:13:50.560837 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:50.540275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38019 (local address 127.0.0.1:46534)
0927 22:13:50.551695 (+ 11420us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:50.551712 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:50.551895 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:50.552733 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:50.552737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:50.552752 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:50.553307 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:50.553318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.555552 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.555560 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:50.557958 (+ 2398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:50.557969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.559983 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.559992 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:50.560295 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:50.560356 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 22:13:50.560446 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 22:13:50.560575 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11297,"mutex_wait_us":36,"thread_start_us":11015,"threads_started":1}
I20260927 22:13:50.561107 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:50.538563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48764 (local address 127.0.97.2:44755)
0927 22:13:50.549527 (+ 10964us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:50.549541 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:50.551363 (+ 1822us) server_negotiation.cc:407] Connection header received
0927 22:13:50.551484 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:50.551495 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:50.551793 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:50.552847 (+ 1054us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:50.553035 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.554539 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.557782 (+ 3243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.559300 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.559441 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:50.559711 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:50.560554 (+ 843us) server_negotiation.cc:299] Negotiation successful
0927 22:13:50.560727 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10727,"thread_start_us":10575,"threads_started":1}
I20260927 22:13:50.561250 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:50.538247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48228 (local address 127.0.97.3:33955)
0927 22:13:50.551284 (+ 13037us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:50.551294 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:50.551907 (+ 613us) server_negotiation.cc:407] Connection header received
0927 22:13:50.552016 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:50.552023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:50.552255 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:50.552437 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:50.553556 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.554740 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.557292 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.558800 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.558952 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:50.560438 (+ 1486us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:50.560521 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 22:13:50.560718 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12824,"thread_start_us":12663,"threads_started":1}
I20260927 22:13:50.561565 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:50.541029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46534 (local address 127.0.97.1:38019)
0927 22:13:50.551243 (+ 10214us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:50.551253 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:50.551790 (+ 537us) server_negotiation.cc:407] Connection header received
0927 22:13:50.552067 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:50.552074 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:50.552253 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:50.552426 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:50.553520 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.555415 (+ 1895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.558136 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.559499 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.559608 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:50.559833 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:50.561072 (+ 1239us) server_negotiation.cc:299] Negotiation successful
0927 22:13:50.561263 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10025,"thread_start_us":9875,"threads_started":1}
I20260927 22:13:50.561831 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:50.538139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44755 (local address 127.0.0.1:48764)
0927 22:13:50.550045 (+ 11906us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:50.550106 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:50.550332 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:50.552131 (+ 1799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:50.552140 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:50.552185 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:50.552654 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:50.552667 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.554707 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.554712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:50.557575 (+ 2863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:50.557589 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.559885 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.559893 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:50.560107 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:50.560144 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:13:50.560436 (+ 292us) client_negotiation.cc:240] Negotiation successful
0927 22:13:50.561417 (+ 981us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11574,"mutex_wait_us":1209,"thread_start_us":14296,"threads_started":1}
I20260927 22:13:50.565215 19974 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc8af51baa544d4a482675cccf7c738 (DEFAULT_TABLE table=test-table [id=f7c05e605d9c4d408f8690ed7d8028a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:13:50.565239 19840 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc8af51baa544d4a482675cccf7c738 (DEFAULT_TABLE table=test-table [id=f7c05e605d9c4d408f8690ed7d8028a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:13:50.565239 20107 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc8af51baa544d4a482675cccf7c738 (DEFAULT_TABLE table=test-table [id=f7c05e605d9c4d408f8690ed7d8028a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:13:50.566500 19974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc8af51baa544d4a482675cccf7c738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:50.566700 20107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc8af51baa544d4a482675cccf7c738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:50.566701 19840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc8af51baa544d4a482675cccf7c738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:50.579182 20190 tablet_bootstrap.cc:492] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a: Bootstrap starting.
I20260927 22:13:50.582005 20191 tablet_bootstrap.cc:492] T 1bc8af51baa544d4a482675cccf7c738 P ef25b78f79114e9282107cf292d523c6: Bootstrap starting.
I20260927 22:13:50.582584 20190 tablet_bootstrap.cc:654] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:50.583575 20192 tablet_bootstrap.cc:492] T 1bc8af51baa544d4a482675cccf7c738 P fb96f7a76df34474aefd99bbc14977ab: Bootstrap starting.
I20260927 22:13:50.583926 20190 log.cc:824] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:50.585724 20191 tablet_bootstrap.cc:654] T 1bc8af51baa544d4a482675cccf7c738 P ef25b78f79114e9282107cf292d523c6: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:50.586573 20192 tablet_bootstrap.cc:654] T 1bc8af51baa544d4a482675cccf7c738 P fb96f7a76df34474aefd99bbc14977ab: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:50.587128 20191 log.cc:824] T 1bc8af51baa544d4a482675cccf7c738 P ef25b78f79114e9282107cf292d523c6: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:50.587234 20190 tablet_bootstrap.cc:492] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a: No bootstrap required, opened a new log
I20260927 22:13:50.587503 20190 ts_tablet_manager.cc:1397] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260927 22:13:50.587803 20192 log.cc:824] T 1bc8af51baa544d4a482675cccf7c738 P fb96f7a76df34474aefd99bbc14977ab: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:50.590834 20191 tablet_bootstrap.cc:492] T 1bc8af51baa544d4a482675cccf7c738 P ef25b78f79114e9282107cf292d523c6: No bootstrap required, opened a new log
I20260927 22:13:50.591091 20191 ts_tablet_manager.cc:1397] T 1bc8af51baa544d4a482675cccf7c738 P ef25b78f79114e9282107cf292d523c6: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 22:13:50.592718 20192 tablet_bootstrap.cc:492] T 1bc8af51baa544d4a482675cccf7c738 P fb96f7a76df34474aefd99bbc14977ab: No bootstrap required, opened a new log
I20260927 22:13:50.592984 20192 ts_tablet_manager.cc:1397] T 1bc8af51baa544d4a482675cccf7c738 P fb96f7a76df34474aefd99bbc14977ab: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260927 22:13:50.597982 20190 raft_consensus.cc:348] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb96f7a76df34474aefd99bbc14977ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33955 } } peers { permanent_uuid: "d9c7b595aade4eca8a2d2268c0a8194a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44755 } } peers { permanent_uuid: "ef25b78f79114e9282107cf292d523c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38019 } }
I20260927 22:13:50.598511 20190 raft_consensus.cc:374] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:50.598649 20190 raft_consensus.cc:729] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9c7b595aade4eca8a2d2268c0a8194a, State: Initialized, Role: FOLLOWER
I20260927 22:13:50.599289 20190 consensus_queue.cc:260] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb96f7a76df34474aefd99bbc14977ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33955 } } peers { permanent_uuid: "d9c7b595aade4eca8a2d2268c0a8194a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44755 } } peers { permanent_uuid: "ef25b78f79114e9282107cf292d523c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38019 } }
I20260927 22:13:50.600539 20192 raft_consensus.cc:348] T 1bc8af51baa544d4a482675cccf7c738 P fb96f7a76df34474aefd99bbc14977ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb96f7a76df34474aefd99bbc14977ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33955 } } peers { permanent_uuid: "d9c7b595aade4eca8a2d2268c0a8194a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44755 } } peers { permanent_uuid: "ef25b78f79114e9282107cf292d523c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38019 } }
I20260927 22:13:50.600878 20192 raft_consensus.cc:374] T 1bc8af51baa544d4a482675cccf7c738 P fb96f7a76df34474aefd99bbc14977ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:50.600991 20192 raft_consensus.cc:729] T 1bc8af51baa544d4a482675cccf7c738 P fb96f7a76df34474aefd99bbc14977ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb96f7a76df34474aefd99bbc14977ab, State: Initialized, Role: FOLLOWER
I20260927 22:13:50.601549 20192 consensus_queue.cc:260] T 1bc8af51baa544d4a482675cccf7c738 P fb96f7a76df34474aefd99bbc14977ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb96f7a76df34474aefd99bbc14977ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33955 } } peers { permanent_uuid: "d9c7b595aade4eca8a2d2268c0a8194a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44755 } } peers { permanent_uuid: "ef25b78f79114e9282107cf292d523c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38019 } }
I20260927 22:13:50.602641 20172 heartbeater.cc:503] Master 127.0.97.62:36509 was elected leader, sending a full tablet report...
I20260927 22:13:50.603200 20192 ts_tablet_manager.cc:1428] T 1bc8af51baa544d4a482675cccf7c738 P fb96f7a76df34474aefd99bbc14977ab: Time spent starting tablet: real 0.010s user 0.011s sys 0.001s
I20260927 22:13:50.603389 20191 raft_consensus.cc:348] T 1bc8af51baa544d4a482675cccf7c738 P ef25b78f79114e9282107cf292d523c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb96f7a76df34474aefd99bbc14977ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33955 } } peers { permanent_uuid: "d9c7b595aade4eca8a2d2268c0a8194a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44755 } } peers { permanent_uuid: "ef25b78f79114e9282107cf292d523c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38019 } }
I20260927 22:13:50.603793 20191 raft_consensus.cc:374] T 1bc8af51baa544d4a482675cccf7c738 P ef25b78f79114e9282107cf292d523c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:50.603947 20191 raft_consensus.cc:729] T 1bc8af51baa544d4a482675cccf7c738 P ef25b78f79114e9282107cf292d523c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef25b78f79114e9282107cf292d523c6, State: Initialized, Role: FOLLOWER
I20260927 22:13:50.604636 20191 consensus_queue.cc:260] T 1bc8af51baa544d4a482675cccf7c738 P ef25b78f79114e9282107cf292d523c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb96f7a76df34474aefd99bbc14977ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33955 } } peers { permanent_uuid: "d9c7b595aade4eca8a2d2268c0a8194a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44755 } } peers { permanent_uuid: "ef25b78f79114e9282107cf292d523c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38019 } }
I20260927 22:13:50.607923 20039 heartbeater.cc:503] Master 127.0.97.62:36509 was elected leader, sending a full tablet report...
I20260927 22:13:50.608896 20190 ts_tablet_manager.cc:1428] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a: Time spent starting tablet: real 0.021s user 0.007s sys 0.012s
I20260927 22:13:50.609467 20191 ts_tablet_manager.cc:1428] T 1bc8af51baa544d4a482675cccf7c738 P ef25b78f79114e9282107cf292d523c6: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
W20260927 22:13:50.681663 19906 tablet.cc:2369] T 1bc8af51baa544d4a482675cccf7c738 P ef25b78f79114e9282107cf292d523c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:13:50.696642 20040 tablet.cc:2369] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:13:50.712433 20173 tablet.cc:2369] T 1bc8af51baa544d4a482675cccf7c738 P fb96f7a76df34474aefd99bbc14977ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:13:50.870201 20196 raft_consensus.cc:482] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:13:50.870575 20196 raft_consensus.cc:504] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb96f7a76df34474aefd99bbc14977ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33955 } } peers { permanent_uuid: "d9c7b595aade4eca8a2d2268c0a8194a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44755 } } peers { permanent_uuid: "ef25b78f79114e9282107cf292d523c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38019 } }
I20260927 22:13:50.872424 20196 leader_election.cc:288] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb96f7a76df34474aefd99bbc14977ab (127.0.97.3:33955), ef25b78f79114e9282107cf292d523c6 (127.0.97.1:38019)
I20260927 22:13:50.879913 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:50.872932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40887 (local address 127.0.97.3:33955)
0927 22:13:50.873244 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:50.873253 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:50.873993 (+ 740us) server_negotiation.cc:407] Connection header received
0927 22:13:50.874183 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:50.874189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:50.874271 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:50.874377 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:50.875176 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.876104 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.878104 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.878914 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.879010 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:50.879087 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:50.879483 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 22:13:50.879621 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260927 22:13:50.879942 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:50.872984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50535 (local address 127.0.97.1:38019)
0927 22:13:50.873222 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:50.873232 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:50.874146 (+ 914us) server_negotiation.cc:407] Connection header received
0927 22:13:50.874389 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:50.874395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:50.874477 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:50.874620 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:50.875383 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.876082 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.878163 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.879034 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.879169 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:50.879274 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:50.879583 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 22:13:50.879723 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:13:50.880054 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:50.872867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38019 (local address 127.0.97.2:50535)
0927 22:13:50.874005 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:50.874023 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:50.874180 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:50.874669 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:50.874677 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:50.874703 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:50.875211 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:50.875224 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.876236 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.876242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:50.877984 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:50.877996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.879260 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.879265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:50.879392 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:50.879446 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 22:13:50.879540 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 22:13:50.879665 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"mutex_wait_us":144,"thread_start_us":585,"threads_started":1}
I20260927 22:13:50.880582 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:50.872823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33955 (local address 127.0.97.2:40887)
0927 22:13:50.873878 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:50.873893 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:50.874031 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:50.874410 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:50.874423 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:50.874446 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:50.875016 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:50.875031 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.876267 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.876271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:50.877944 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:50.877956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.879130 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.879137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:50.879323 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:50.879352 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 22:13:50.879431 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 22:13:50.879559 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"mutex_wait_us":28,"thread_start_us":740,"threads_started":1}
I20260927 22:13:50.881307 19860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc8af51baa544d4a482675cccf7c738" candidate_uuid: "d9c7b595aade4eca8a2d2268c0a8194a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef25b78f79114e9282107cf292d523c6" is_pre_election: true
I20260927 22:13:50.881518 20127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc8af51baa544d4a482675cccf7c738" candidate_uuid: "d9c7b595aade4eca8a2d2268c0a8194a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb96f7a76df34474aefd99bbc14977ab" is_pre_election: true
I20260927 22:13:50.881888 19860 raft_consensus.cc:2454] T 1bc8af51baa544d4a482675cccf7c738 P ef25b78f79114e9282107cf292d523c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9c7b595aade4eca8a2d2268c0a8194a in term 0.
I20260927 22:13:50.882058 20127 raft_consensus.cc:2454] T 1bc8af51baa544d4a482675cccf7c738 P fb96f7a76df34474aefd99bbc14977ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9c7b595aade4eca8a2d2268c0a8194a in term 0.
I20260927 22:13:50.882573 19929 leader_election.cc:302] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9c7b595aade4eca8a2d2268c0a8194a, ef25b78f79114e9282107cf292d523c6; no voters:
I20260927 22:13:50.883023 20196 raft_consensus.cc:2781] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:13:50.883164 20196 raft_consensus.cc:482] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:13:50.883287 20196 raft_consensus.cc:3037] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:50.885525 20196 raft_consensus.cc:504] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb96f7a76df34474aefd99bbc14977ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33955 } } peers { permanent_uuid: "d9c7b595aade4eca8a2d2268c0a8194a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44755 } } peers { permanent_uuid: "ef25b78f79114e9282107cf292d523c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38019 } }
I20260927 22:13:50.886217 20196 leader_election.cc:288] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a [CANDIDATE]: Term 1 election: Requested vote from peers fb96f7a76df34474aefd99bbc14977ab (127.0.97.3:33955), ef25b78f79114e9282107cf292d523c6 (127.0.97.1:38019)
I20260927 22:13:50.886786 20127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc8af51baa544d4a482675cccf7c738" candidate_uuid: "d9c7b595aade4eca8a2d2268c0a8194a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb96f7a76df34474aefd99bbc14977ab"
I20260927 22:13:50.886930 19860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc8af51baa544d4a482675cccf7c738" candidate_uuid: "d9c7b595aade4eca8a2d2268c0a8194a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef25b78f79114e9282107cf292d523c6"
I20260927 22:13:50.887015 20127 raft_consensus.cc:3037] T 1bc8af51baa544d4a482675cccf7c738 P fb96f7a76df34474aefd99bbc14977ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:50.887178 19860 raft_consensus.cc:3037] T 1bc8af51baa544d4a482675cccf7c738 P ef25b78f79114e9282107cf292d523c6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:50.889251 20127 raft_consensus.cc:2454] T 1bc8af51baa544d4a482675cccf7c738 P fb96f7a76df34474aefd99bbc14977ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9c7b595aade4eca8a2d2268c0a8194a in term 1.
I20260927 22:13:50.889408 19860 raft_consensus.cc:2454] T 1bc8af51baa544d4a482675cccf7c738 P ef25b78f79114e9282107cf292d523c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9c7b595aade4eca8a2d2268c0a8194a in term 1.
I20260927 22:13:50.889679 19928 leader_election.cc:302] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9c7b595aade4eca8a2d2268c0a8194a, fb96f7a76df34474aefd99bbc14977ab; no voters:
I20260927 22:13:50.890007 20196 raft_consensus.cc:2781] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:13:50.890512 20196 raft_consensus.cc:686] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a [term 1 LEADER]: Becoming Leader. State: Replica: d9c7b595aade4eca8a2d2268c0a8194a, State: Running, Role: LEADER
I20260927 22:13:50.891068 20196 consensus_queue.cc:237] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb96f7a76df34474aefd99bbc14977ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33955 } } peers { permanent_uuid: "d9c7b595aade4eca8a2d2268c0a8194a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44755 } } peers { permanent_uuid: "ef25b78f79114e9282107cf292d523c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38019 } }
I20260927 22:13:50.895756 19716 catalog_manager.cc:5495] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a reported cstate change: term changed from 0 to 1, leader changed from <none> to d9c7b595aade4eca8a2d2268c0a8194a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d9c7b595aade4eca8a2d2268c0a8194a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb96f7a76df34474aefd99bbc14977ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9c7b595aade4eca8a2d2268c0a8194a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef25b78f79114e9282107cf292d523c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38019 } health_report { overall_health: UNKNOWN } } }
I20260927 22:13:50.925021 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:50.919446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48772 (local address 127.0.97.2:44755)
0927 22:13:50.919634 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:50.919639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:50.919712 (+ 73us) server_negotiation.cc:407] Connection header received
0927 22:13:50.919948 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:50.919952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:50.920055 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:50.920227 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:13:50.920530 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.921328 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.922794 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:50.923197 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:50.923332 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:50.923876 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:13:50.924412 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:13:50.924576 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:50.924635 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 22:13:50.924715 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:13:50.933430 19860 raft_consensus.cc:1261] T 1bc8af51baa544d4a482675cccf7c738 P ef25b78f79114e9282107cf292d523c6 [term 1 FOLLOWER]: Refusing update from remote peer d9c7b595aade4eca8a2d2268c0a8194a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:13:50.933429 20127 raft_consensus.cc:1261] T 1bc8af51baa544d4a482675cccf7c738 P fb96f7a76df34474aefd99bbc14977ab [term 1 FOLLOWER]: Refusing update from remote peer d9c7b595aade4eca8a2d2268c0a8194a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:13:50.934450 20201 consensus_queue.cc:1035] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb96f7a76df34474aefd99bbc14977ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:13:50.934950 20196 consensus_queue.cc:1035] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef25b78f79114e9282107cf292d523c6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:13:50.946549 20204 mvcc.cc:205] Tried to move back new op lower bound from 7334081457892519936 to 7334081457733816320. Current Snapshot: MvccSnapshot[applied={T|T < 7334081457892519936}]
I20260927 22:13:50.950441 20205 mvcc.cc:205] Tried to move back new op lower bound from 7334081457892519936 to 7334081457733816320. Current Snapshot: MvccSnapshot[applied={T|T < 7334081457892519936}]
I20260927 22:13:50.962757 20206 mvcc.cc:205] Tried to move back new op lower bound from 7334081457892519936 to 7334081457733816320. Current Snapshot: MvccSnapshot[applied={T|T < 7334081457892519936}]
I20260927 22:13:50.978878 19716 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:13:50.979141 19716 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:13:50.982630 19716 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17698107cf1b410d91bb5f1bb9b49b2b: Sending DeleteTablet for 3 replicas of tablet 1bc8af51baa544d4a482675cccf7c738
I20260927 22:13:50.984050 19974 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bc8af51baa544d4a482675cccf7c738 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:13:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48764
I20260927 22:13:50.984726 19840 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bc8af51baa544d4a482675cccf7c738 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:13:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46534
I20260927 22:13:50.983783 20107 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bc8af51baa544d4a482675cccf7c738 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:13:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48228
I20260927 22:13:50.987087 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 19777
I20260927 22:13:50.994542 20215 tablet_replica.cc:330] T 1bc8af51baa544d4a482675cccf7c738 P ef25b78f79114e9282107cf292d523c6: stopping tablet replica
I20260927 22:13:50.995062 20215 raft_consensus.cc:2229] T 1bc8af51baa544d4a482675cccf7c738 P ef25b78f79114e9282107cf292d523c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:13:50.995620 20215 raft_consensus.cc:2258] T 1bc8af51baa544d4a482675cccf7c738 P ef25b78f79114e9282107cf292d523c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:13:51.002431 20214 tablet_replica.cc:330] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a: stopping tablet replica
I20260927 22:13:51.002651 20216 tablet_replica.cc:330] T 1bc8af51baa544d4a482675cccf7c738 P fb96f7a76df34474aefd99bbc14977ab: stopping tablet replica
I20260927 22:13:51.002956 20214 raft_consensus.cc:2229] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:13:51.003665 20214 raft_consensus.cc:2258] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:13:51.004069 20216 raft_consensus.cc:2229] T 1bc8af51baa544d4a482675cccf7c738 P fb96f7a76df34474aefd99bbc14977ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:13:51.004570 20216 raft_consensus.cc:2258] T 1bc8af51baa544d4a482675cccf7c738 P fb96f7a76df34474aefd99bbc14977ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:13:51.005389 20215 ts_tablet_manager.cc:1905] T 1bc8af51baa544d4a482675cccf7c738 P ef25b78f79114e9282107cf292d523c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:13:51.006726 20216 ts_tablet_manager.cc:1905] T 1bc8af51baa544d4a482675cccf7c738 P fb96f7a76df34474aefd99bbc14977ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:13:51.009895 20214 ts_tablet_manager.cc:1905] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:13:51.012929 20215 ts_tablet_manager.cc:1918] T 1bc8af51baa544d4a482675cccf7c738 P ef25b78f79114e9282107cf292d523c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:13:51.013103 20215 log.cc:1196] T 1bc8af51baa544d4a482675cccf7c738 P ef25b78f79114e9282107cf292d523c6: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-0/wal/wals/1bc8af51baa544d4a482675cccf7c738
I20260927 22:13:51.013648 20215 ts_tablet_manager.cc:1939] T 1bc8af51baa544d4a482675cccf7c738 P ef25b78f79114e9282107cf292d523c6: Deleting consensus metadata
I20260927 22:13:51.013710 20216 ts_tablet_manager.cc:1918] T 1bc8af51baa544d4a482675cccf7c738 P fb96f7a76df34474aefd99bbc14977ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:13:51.013867 20216 log.cc:1196] T 1bc8af51baa544d4a482675cccf7c738 P fb96f7a76df34474aefd99bbc14977ab: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/1bc8af51baa544d4a482675cccf7c738
I20260927 22:13:51.014340 20216 ts_tablet_manager.cc:1939] T 1bc8af51baa544d4a482675cccf7c738 P fb96f7a76df34474aefd99bbc14977ab: Deleting consensus metadata
I20260927 22:13:51.015599 19702 catalog_manager.cc:4841] TS ef25b78f79114e9282107cf292d523c6 (127.0.97.1:38019): tablet 1bc8af51baa544d4a482675cccf7c738 (table test-table [id=f7c05e605d9c4d408f8690ed7d8028a5]) successfully deleted
I20260927 22:13:51.016414 19701 catalog_manager.cc:4841] TS fb96f7a76df34474aefd99bbc14977ab (127.0.97.3:33955): tablet 1bc8af51baa544d4a482675cccf7c738 (table test-table [id=f7c05e605d9c4d408f8690ed7d8028a5]) successfully deleted
Sep 27 22:13:51 dist-test-slave-mp03 krb5kdc[19671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:13:51.020814 20214 ts_tablet_manager.cc:1918] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:13:51.021111 20214 log.cc:1196] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/1bc8af51baa544d4a482675cccf7c738
I20260927 22:13:51.021679 20214 ts_tablet_manager.cc:1939] T 1bc8af51baa544d4a482675cccf7c738 P d9c7b595aade4eca8a2d2268c0a8194a: Deleting consensus metadata
I20260927 22:13:51.023494 19703 catalog_manager.cc:4841] TS d9c7b595aade4eca8a2d2268c0a8194a (127.0.97.2:44755): tablet 1bc8af51baa544d4a482675cccf7c738 (table test-table [id=f7c05e605d9c4d408f8690ed7d8028a5]) successfully deleted
I20260927 22:13:51.027418 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:50.989375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46544 (local address 127.0.97.1:38019)
0927 22:13:50.989628 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:50.989650 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:51.001568 (+ 11918us) server_negotiation.cc:407] Connection header received
0927 22:13:51.002140 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:51.002148 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:51.002292 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:51.002393 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:51.006755 (+ 4362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:51.007940 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:51.013709 (+ 5769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:51.014358 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:51.014459 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:51.020793 (+ 6334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:51.020810 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:51.020831 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:51.020878 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:51.024643 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:51.025354 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:51.025362 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:51.025364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:51.025490 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:51.025887 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:51.025899 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:51.025901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:51.026222 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:51.026524 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:51.026847 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 22:13:51.027072 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:13:51.028024 19900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:51.161832 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 19777
I20260927 22:13:51.180169 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 19911
Sep 27 22:13:51 dist-test-slave-mp03 krb5kdc[19671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:13:51.198011 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:51.180935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48780 (local address 127.0.97.2:44755)
0927 22:13:51.181128 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:51.181133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:51.181154 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:13:51.181347 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:51.181350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:51.181420 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:51.181503 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:51.183165 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:51.183947 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:51.185853 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:51.186280 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:51.186367 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:51.191441 (+ 5074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:51.191455 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:51.191472 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:51.191513 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:51.194971 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:51.195742 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:51.195751 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:51.195753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:51.195907 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:51.196320 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:51.196327 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:51.196330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:51.196702 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:51.197118 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:51.197345 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 22:13:51.197575 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 22:13:51.198767 20034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:51.311453 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 19911
I20260927 22:13:51.330962 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 20044
Sep 27 22:13:51 dist-test-slave-mp03 krb5kdc[19671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:13:51.346882 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:51.331751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48242 (local address 127.0.97.3:33955)
0927 22:13:51.331934 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:51.331940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:51.332051 (+ 111us) server_negotiation.cc:407] Connection header received
0927 22:13:51.332251 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:51.332263 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:51.332330 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:51.332441 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:51.333732 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:51.334498 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:51.336336 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:51.337021 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:51.337184 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:51.341774 (+ 4590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:51.341787 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:51.341804 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:51.341843 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:51.344475 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:51.345089 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:51.345096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:51.345098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:51.345225 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:51.345522 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:51.345524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:51.345526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:51.345860 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:51.346073 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:51.346306 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 22:13:51.346464 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:13:51.347540 20167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:51.436887 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 20044
I20260927 22:13:51.453703 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 19684
I20260927 22:13:51.454792 19746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:51.566205 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 19684
2026-09-27T22:13:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4637 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T22:13:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:13:51Z Disabled control of system clock
I20260927 22:13:51.643105 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38197
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:13:51.977735 20225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:51.978050 20225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:51.978134 20225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:51.985476 20225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:13:51.985597 20225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:51.985635 20225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:51.985687 20225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:13:51.985731 20225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:13:51.994979 20225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38197
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:51.996337 20225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:51.998072 20225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:52.010213 20231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:52.010164 20230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:52.010545 20233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:52.011432 20225 server_base.cc:1034] running on GCE node
I20260927 22:13:52.012208 20225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:52.013667 20225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:52.014880 20225 hybrid_clock.cc:648] HybridClock initialized: now 1790547232014797 us; error 82 us; skew 500 ppm
I20260927 22:13:52.015257 20225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:13:52.017645 20225 webserver.cc:460] Webserver started at http://127.0.97.62:42735/ using document root <none> and password file <none>
I20260927 22:13:52.018342 20225 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:52.018472 20225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:52.018874 20225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:52.021723 20225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "4c089c5f81f141d6b0fce05ff2700f15"
format_stamp: "Formatted at 2026-09-27 22:13:52 on dist-test-slave-mp03"
server_key: "e0d28fceade9263f4119aad5f8999741"
server_key_iv: "3475a2b9d759958da0b37832753b3d75"
server_key_version: "encryptionkey@0"
I20260927 22:13:52.022534 20225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "4c089c5f81f141d6b0fce05ff2700f15"
format_stamp: "Formatted at 2026-09-27 22:13:52 on dist-test-slave-mp03"
server_key: "e0d28fceade9263f4119aad5f8999741"
server_key_iv: "3475a2b9d759958da0b37832753b3d75"
server_key_version: "encryptionkey@0"
I20260927 22:13:52.027745 20225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 22:13:52.031268 20239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:52.032646 20225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:13:52.032831 20225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "4c089c5f81f141d6b0fce05ff2700f15"
format_stamp: "Formatted at 2026-09-27 22:13:52 on dist-test-slave-mp03"
server_key: "e0d28fceade9263f4119aad5f8999741"
server_key_iv: "3475a2b9d759958da0b37832753b3d75"
server_key_version: "encryptionkey@0"
I20260927 22:13:52.033071 20225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:52.045426 20225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:52.046222 20225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:52.046505 20225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:52.065320 20225 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38197
I20260927 22:13:52.065341 20290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38197 every 8 connection(s)
I20260927 22:13:52.066744 20225 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:13:52.070398 20291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:52.077003 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 20225
I20260927 22:13:52.077178 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:13:52.077622 388 external_mini_cluster.cc:1428] Setting key caf8a5e487c30c156b3380ffd2b3bd6b
I20260927 22:13:52.081377 20291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c089c5f81f141d6b0fce05ff2700f15: Bootstrap starting.
I20260927 22:13:52.084483 20291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c089c5f81f141d6b0fce05ff2700f15: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:52.085500 20291 log.cc:824] T 00000000000000000000000000000000 P 4c089c5f81f141d6b0fce05ff2700f15: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:52.088195 20291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c089c5f81f141d6b0fce05ff2700f15: No bootstrap required, opened a new log
I20260927 22:13:52.091039 20294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:52.079613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56156 (local address 127.0.97.62:38197)
0927 22:13:52.080562 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:52.080573 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:52.080618 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:13:52.082355 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:52.082377 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:52.082904 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:52.083319 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:52.083841 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:52.085176 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:52.086576 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:52.087331 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:52.087533 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:52.088680 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:52.088690 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:52.088709 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:52.089250 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:52.089347 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:52.089367 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:52.089500 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:52.089734 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 22:13:52.090034 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":315,"threads_started":1}
I20260927 22:13:52.095788 20291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c089c5f81f141d6b0fce05ff2700f15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c089c5f81f141d6b0fce05ff2700f15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38197 } }
I20260927 22:13:52.096141 20291 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c089c5f81f141d6b0fce05ff2700f15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:52.096246 20291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c089c5f81f141d6b0fce05ff2700f15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c089c5f81f141d6b0fce05ff2700f15, State: Initialized, Role: FOLLOWER
I20260927 22:13:52.096760 20291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c089c5f81f141d6b0fce05ff2700f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c089c5f81f141d6b0fce05ff2700f15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38197 } }
I20260927 22:13:52.097039 20291 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c089c5f81f141d6b0fce05ff2700f15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:13:52.097211 20291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c089c5f81f141d6b0fce05ff2700f15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:13:52.097385 20291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c089c5f81f141d6b0fce05ff2700f15 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:52.099962 20291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c089c5f81f141d6b0fce05ff2700f15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c089c5f81f141d6b0fce05ff2700f15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38197 } }
I20260927 22:13:52.100440 20291 leader_election.cc:302] T 00000000000000000000000000000000 P 4c089c5f81f141d6b0fce05ff2700f15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c089c5f81f141d6b0fce05ff2700f15; no voters:
I20260927 22:13:52.100966 20291 leader_election.cc:288] T 00000000000000000000000000000000 P 4c089c5f81f141d6b0fce05ff2700f15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:13:52.101091 20296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c089c5f81f141d6b0fce05ff2700f15 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:13:52.101850 20296 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c089c5f81f141d6b0fce05ff2700f15 [term 1 LEADER]: Becoming Leader. State: Replica: 4c089c5f81f141d6b0fce05ff2700f15, State: Running, Role: LEADER
I20260927 22:13:52.102381 20296 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c089c5f81f141d6b0fce05ff2700f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c089c5f81f141d6b0fce05ff2700f15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38197 } }
I20260927 22:13:52.103176 20291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c089c5f81f141d6b0fce05ff2700f15 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:13:52.105178 20298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c089c5f81f141d6b0fce05ff2700f15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c089c5f81f141d6b0fce05ff2700f15. Latest consensus state: current_term: 1 leader_uuid: "4c089c5f81f141d6b0fce05ff2700f15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c089c5f81f141d6b0fce05ff2700f15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38197 } } }
I20260927 22:13:52.105291 20297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c089c5f81f141d6b0fce05ff2700f15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c089c5f81f141d6b0fce05ff2700f15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c089c5f81f141d6b0fce05ff2700f15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38197 } } }
I20260927 22:13:52.105540 20298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c089c5f81f141d6b0fce05ff2700f15 [sys.catalog]: This master's current role is: LEADER
I20260927 22:13:52.105604 20297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c089c5f81f141d6b0fce05ff2700f15 [sys.catalog]: This master's current role is: LEADER
I20260927 22:13:52.107113 20301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:13:52.113565 20301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:13:52.123953 20301 catalog_manager.cc:1347] Generated new cluster ID: 4c744a3613914de78e5eb85975275ffb
I20260927 22:13:52.124078 20301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:13:52.143900 20301 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:13:52.145282 20301 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:13:52.153861 20301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c089c5f81f141d6b0fce05ff2700f15: Generated new TSK 0
I20260927 22:13:52.154800 20301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:13:52.164121 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38197
--builtin_ntp_servers=127.0.97.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:13:52.520167 20315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:52.520448 20315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:52.520535 20315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:52.527810 20315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:52.527909 20315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:52.528023 20315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:13:52.537586 20315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:52.538923 20315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:52.540616 20315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:52.554075 20321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:52.554032 20323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:52.554117 20320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:52.555066 20315 server_base.cc:1034] running on GCE node
I20260927 22:13:52.555706 20315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:52.556649 20315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:52.557955 20315 hybrid_clock.cc:648] HybridClock initialized: now 1790547232557848 us; error 81 us; skew 500 ppm
I20260927 22:13:52.558311 20315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:13:52.560678 20315 webserver.cc:460] Webserver started at http://127.0.97.1:35539/ using document root <none> and password file <none>
I20260927 22:13:52.561380 20315 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:52.561510 20315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:52.561842 20315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:52.564730 20315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-0/data/instance:
uuid: "da1069de647b4c538b111246e441df95"
format_stamp: "Formatted at 2026-09-27 22:13:52 on dist-test-slave-mp03"
server_key: "d8872f85befee03530ae1315806a539e"
server_key_iv: "5a877173cc5bb0053694e04a9ea5668b"
server_key_version: "encryptionkey@0"
I20260927 22:13:52.565516 20315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-0/wal/instance:
uuid: "da1069de647b4c538b111246e441df95"
format_stamp: "Formatted at 2026-09-27 22:13:52 on dist-test-slave-mp03"
server_key: "d8872f85befee03530ae1315806a539e"
server_key_iv: "5a877173cc5bb0053694e04a9ea5668b"
server_key_version: "encryptionkey@0"
I20260927 22:13:52.570568 20315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 22:13:52.574050 20329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:52.575462 20315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:13:52.575635 20315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-0/wal
uuid: "da1069de647b4c538b111246e441df95"
format_stamp: "Formatted at 2026-09-27 22:13:52 on dist-test-slave-mp03"
server_key: "d8872f85befee03530ae1315806a539e"
server_key_iv: "5a877173cc5bb0053694e04a9ea5668b"
server_key_version: "encryptionkey@0"
I20260927 22:13:52.575882 20315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:52.589010 20315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:52.589809 20315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:52.590093 20315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:52.591063 20315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:13:52.592478 20315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:13:52.592605 20315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:52.592731 20315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:13:52.592818 20315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:52.625397 20315 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42857
I20260927 22:13:52.625432 20441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42857 every 8 connection(s)
I20260927 22:13:52.626971 20315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:13:52.636416 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 20315
I20260927 22:13:52.636606 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:13:52.637212 388 external_mini_cluster.cc:1428] Setting key f2ad05af94d4ca1f1a84393faa4079b4
I20260927 22:13:52.640691 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38197
--builtin_ntp_servers=127.0.97.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:13:52.641235 20444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:52.629744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50179 (local address 127.0.97.62:38197)
0927 22:13:52.630157 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:52.630162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:52.631597 (+ 1435us) server_negotiation.cc:407] Connection header received
0927 22:13:52.633420 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:52.633425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:52.633532 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:52.633634 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:52.634827 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:52.635577 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:52.638069 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:52.638565 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:52.638742 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:52.640350 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:52.640355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:52.640358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:52.640375 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:52.640419 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:52.640431 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:52.640515 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:52.640806 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 22:13:52.640924 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":308,"thread_start_us":210,"threads_started":1}
I20260927 22:13:52.642797 20445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:52.630408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38197 (local address 127.0.97.1:50179)
0927 22:13:52.631428 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:52.631477 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:52.633139 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:52.633840 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:52.633848 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:52.633973 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:13:52.634561 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:52.634577 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:52.635744 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:52.635748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:52.637922 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:52.637936 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:52.638869 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:52.638878 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:52.639188 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:52.639954 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:13:52.639958 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:13:52.640013 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:13:52.640139 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:13:52.640542 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:13:52.640555 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:13:52.640563 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:13:52.640787 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 22:13:52.641256 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":392,"threads_started":1}
I20260927 22:13:52.644398 20442 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38197
I20260927 22:13:52.644701 20442 heartbeater.cc:464] Registering TS with master...
I20260927 22:13:52.645450 20442 heartbeater.cc:511] Master 127.0.97.62:38197 requested a full tablet report, sending...
I20260927 22:13:52.647621 20256 ts_manager.cc:194] Registered new tserver with Master: da1069de647b4c538b111246e441df95 (127.0.97.1:42857)
I20260927 22:13:52.649149 20256 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:50179
W20260927 22:13:52.972208 20446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:52.972502 20446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:52.972640 20446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:52.980175 20446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:52.980274 20446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:52.980459 20446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:13:52.989725 20446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:52.991075 20446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:52.992858 20446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:53.006422 20454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:53.006512 20451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:53.006376 20452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:53.012151 20446 server_base.cc:1034] running on GCE node
I20260927 22:13:53.012797 20446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:53.013784 20446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:53.015038 20446 hybrid_clock.cc:648] HybridClock initialized: now 1790547233014946 us; error 63 us; skew 500 ppm
I20260927 22:13:53.015363 20446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:13:53.017720 20446 webserver.cc:460] Webserver started at http://127.0.97.2:39043/ using document root <none> and password file <none>
I20260927 22:13:53.018404 20446 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:53.018529 20446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:53.018919 20446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:53.021790 20446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-1/data/instance:
uuid: "63fca6ed497542dc864e980c7194474a"
format_stamp: "Formatted at 2026-09-27 22:13:53 on dist-test-slave-mp03"
server_key: "92e5ffaf5276a8b566e2f4e2fb6c3eab"
server_key_iv: "6140e1f0b560817685ea8b64cc5186a8"
server_key_version: "encryptionkey@0"
I20260927 22:13:53.022565 20446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-1/wal/instance:
uuid: "63fca6ed497542dc864e980c7194474a"
format_stamp: "Formatted at 2026-09-27 22:13:53 on dist-test-slave-mp03"
server_key: "92e5ffaf5276a8b566e2f4e2fb6c3eab"
server_key_iv: "6140e1f0b560817685ea8b64cc5186a8"
server_key_version: "encryptionkey@0"
I20260927 22:13:53.027753 20446 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:13:53.031207 20460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:53.032647 20446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:13:53.032828 20446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-1/wal
uuid: "63fca6ed497542dc864e980c7194474a"
format_stamp: "Formatted at 2026-09-27 22:13:53 on dist-test-slave-mp03"
server_key: "92e5ffaf5276a8b566e2f4e2fb6c3eab"
server_key_iv: "6140e1f0b560817685ea8b64cc5186a8"
server_key_version: "encryptionkey@0"
I20260927 22:13:53.033107 20446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:53.045718 20446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:53.046536 20446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:53.046991 20446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:53.048015 20446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:13:53.049495 20446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:13:53.049616 20446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:53.049753 20446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:13:53.049840 20446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:53.093019 20446 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43587
I20260927 22:13:53.093093 20572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43587 every 8 connection(s)
I20260927 22:13:53.094693 20446 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:13:53.098179 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 20446
I20260927 22:13:53.098347 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:13:53.098841 388 external_mini_cluster.cc:1428] Setting key b8cfd585785c829f4cc8dec8d1461481
I20260927 22:13:53.101658 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38197
--builtin_ntp_servers=127.0.97.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:13:53.109232 20444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:53.097551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32803 (local address 127.0.97.62:38197)
0927 22:13:53.097782 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:53.097788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:53.099552 (+ 1764us) server_negotiation.cc:407] Connection header received
0927 22:13:53.101604 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:53.101610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:53.101698 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:53.101815 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:53.103071 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:53.104229 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:53.106011 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:53.106792 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:53.106938 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:53.108438 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:53.108443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:53.108447 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:53.108466 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:53.108499 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:53.108511 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:53.108606 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:53.108831 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 22:13:53.108933 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 22:13:53.110322 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:53.098129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38197 (local address 127.0.97.2:32803)
0927 22:13:53.099349 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:53.099414 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:53.101322 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:53.102100 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:53.102109 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:53.102201 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:13:53.102901 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:53.102922 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:53.104378 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:53.104382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:53.105852 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:53.105865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:53.107040 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:53.107046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:53.107246 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:53.108061 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:13:53.108064 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:13:53.108133 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:13:53.108222 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:13:53.108607 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:13:53.108621 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:13:53.108632 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 22:13:53.108822 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 22:13:53.109235 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":449,"threads_started":1}
I20260927 22:13:53.111975 20573 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38197
I20260927 22:13:53.112293 20573 heartbeater.cc:464] Registering TS with master...
I20260927 22:13:53.113030 20573 heartbeater.cc:511] Master 127.0.97.62:38197 requested a full tablet report, sending...
I20260927 22:13:53.114465 20256 ts_manager.cc:194] Registered new tserver with Master: 63fca6ed497542dc864e980c7194474a (127.0.97.2:43587)
I20260927 22:13:53.115769 20256 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:32803
W20260927 22:13:53.433789 20576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:53.434087 20576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:53.434175 20576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:53.441701 20576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:53.441805 20576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:53.441911 20576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:13:53.451097 20576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:53.452353 20576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:53.454059 20576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:53.467476 20581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:53.467550 20582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:53.467446 20584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:53.472299 20576 server_base.cc:1034] running on GCE node
I20260927 22:13:53.472956 20576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:53.474009 20576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:53.475250 20576 hybrid_clock.cc:648] HybridClock initialized: now 1790547233475186 us; error 63 us; skew 500 ppm
I20260927 22:13:53.475592 20576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:13:53.478063 20576 webserver.cc:460] Webserver started at http://127.0.97.3:43777/ using document root <none> and password file <none>
I20260927 22:13:53.478858 20576 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:53.478989 20576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:53.479341 20576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:53.482349 20576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-2/data/instance:
uuid: "c0e270745a0c4a4bbc2369f2d994753e"
format_stamp: "Formatted at 2026-09-27 22:13:53 on dist-test-slave-mp03"
server_key: "179a6c48d4b8586330f99c3ba136428e"
server_key_iv: "19a4ec335d6dee13cce5d062a705e894"
server_key_version: "encryptionkey@0"
I20260927 22:13:53.483201 20576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c0e270745a0c4a4bbc2369f2d994753e"
format_stamp: "Formatted at 2026-09-27 22:13:53 on dist-test-slave-mp03"
server_key: "179a6c48d4b8586330f99c3ba136428e"
server_key_iv: "19a4ec335d6dee13cce5d062a705e894"
server_key_version: "encryptionkey@0"
I20260927 22:13:53.488397 20576 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:13:53.492069 20590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:53.493455 20576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:13:53.493636 20576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-2/wal
uuid: "c0e270745a0c4a4bbc2369f2d994753e"
format_stamp: "Formatted at 2026-09-27 22:13:53 on dist-test-slave-mp03"
server_key: "179a6c48d4b8586330f99c3ba136428e"
server_key_iv: "19a4ec335d6dee13cce5d062a705e894"
server_key_version: "encryptionkey@0"
I20260927 22:13:53.493860 20576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:53.519817 20576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:53.520687 20576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:53.520975 20576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:53.522008 20576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:13:53.523577 20576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:13:53.523698 20576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:53.523830 20576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:13:53.523921 20576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:53.557816 20576 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37319
I20260927 22:13:53.557904 20702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37319 every 8 connection(s)
I20260927 22:13:53.559677 20576 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:13:53.569234 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 20576
I20260927 22:13:53.569393 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:13:53.569923 388 external_mini_cluster.cc:1428] Setting key 3db04662fe9272491ad3b6118b1c68a4
I20260927 22:13:53.573427 20444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:53.562499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56945 (local address 127.0.97.62:38197)
0927 22:13:53.562741 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:53.562746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:53.564155 (+ 1409us) server_negotiation.cc:407] Connection header received
0927 22:13:53.565935 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:53.565940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:53.566035 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:53.566148 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:53.567284 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:53.568013 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:53.569983 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:53.570499 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:53.570680 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:53.572435 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:53.572443 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:53.572448 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:53.572479 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:53.572530 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:53.572553 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:53.572665 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:53.572923 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 22:13:53.573027 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:13:53.574232 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:53.563053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38197 (local address 127.0.97.3:56945)
0927 22:13:53.564004 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:53.564040 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:53.565672 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:53.566374 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:53.566382 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:53.566476 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:13:53.567119 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:53.567135 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:53.568170 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:53.568175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:53.569817 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:53.569830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:53.570819 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:53.570827 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:53.571171 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:53.572006 (+ 835us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:13:53.572010 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:13:53.572078 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:13:53.572183 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:13:53.572673 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:13:53.572681 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:13:53.572688 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:13:53.572832 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 22:13:53.573192 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":290,"threads_started":1}
I20260927 22:13:53.575500 20703 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38197
I20260927 22:13:53.575801 20703 heartbeater.cc:464] Registering TS with master...
I20260927 22:13:53.576598 20703 heartbeater.cc:511] Master 127.0.97.62:38197 requested a full tablet report, sending...
I20260927 22:13:53.578119 20256 ts_manager.cc:194] Registered new tserver with Master: c0e270745a0c4a4bbc2369f2d994753e (127.0.97.3:37319)
I20260927 22:13:53.578981 20256 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:56945
I20260927 22:13:53.586077 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:13:53.596401 20444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:53.590230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56172 (local address 127.0.97.62:38197)
0927 22:13:53.590470 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:53.590481 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:53.590978 (+ 497us) server_negotiation.cc:407] Connection header received
0927 22:13:53.591094 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:53.591098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:53.591180 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:53.591269 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:53.591733 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:53.592440 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:53.593848 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:53.594308 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:53.594425 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:53.595649 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:53.595654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:53.595658 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:53.595676 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:53.595710 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:53.595721 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:53.595801 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:53.595969 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 22:13:53.596057 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:13:53.600466 20256 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:13:53.607290 20256 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:13:53.633003 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:53.620204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57382 (local address 127.0.97.1:42857)
0927 22:13:53.620964 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:53.620977 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:53.621675 (+ 698us) server_negotiation.cc:407] Connection header received
0927 22:13:53.622082 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:53.622093 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:53.622363 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:53.622541 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:53.623552 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:53.625448 (+ 1896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:53.628898 (+ 3450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:53.630903 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:53.631033 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:53.631400 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:53.632235 (+ 835us) server_negotiation.cc:299] Negotiation successful
0927 22:13:53.632492 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":377,"threads_started":1}
I20260927 22:13:53.634483 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:53.619924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42857 (local address 127.0.0.1:57382)
0927 22:13:53.621533 (+ 1609us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:53.621574 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:53.621829 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:53.622889 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:53.622899 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:53.622946 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:53.623336 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:53.623352 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:53.625652 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:53.625661 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:53.628632 (+ 2971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:53.628652 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:53.631616 (+ 2964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:53.631625 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:53.631918 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:53.631995 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 22:13:53.633770 (+ 1775us) client_negotiation.cc:240] Negotiation successful
0927 22:13:53.634037 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1371,"thread_start_us":1235,"threads_started":1}
I20260927 22:13:53.640764 20377 tablet_service.cc:1467] Processing CreateTablet for tablet 3498bce12376493cb87d4bf60fbfa68d (DEFAULT_TABLE table=test-table [id=a90306dad6814f4f8a4b3e2193b46f45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:13:53.642536 20377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3498bce12376493cb87d4bf60fbfa68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:53.645051 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:53.621966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37319 (local address 127.0.0.1:43098)
0927 22:13:53.636289 (+ 14323us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:53.636305 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:53.636428 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:53.637525 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:53.637529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:53.637558 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:53.637989 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:53.638002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:53.639984 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:53.639989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:53.642068 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:53.642086 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:53.644264 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:53.644271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:53.644469 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:53.644503 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:13:53.644601 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 22:13:53.644745 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14177,"thread_start_us":14036,"threads_started":1}
I20260927 22:13:53.645839 20720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:53.622546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43098 (local address 127.0.97.3:37319)
0927 22:13:53.635808 (+ 13262us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:53.635821 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:53.636567 (+ 746us) server_negotiation.cc:407] Connection header received
0927 22:13:53.636688 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:53.636696 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:53.637037 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:53.637225 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:53.638213 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:53.639814 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:53.642304 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:53.643713 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:53.643855 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:53.644127 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:53.645289 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0927 22:13:53.645477 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13067,"thread_start_us":12871,"threads_started":1}
I20260927 22:13:53.650179 20638 tablet_service.cc:1467] Processing CreateTablet for tablet 3498bce12376493cb87d4bf60fbfa68d (DEFAULT_TABLE table=test-table [id=a90306dad6814f4f8a4b3e2193b46f45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:13:53.651759 20638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3498bce12376493cb87d4bf60fbfa68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:53.652843 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:53.620329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43587 (local address 127.0.0.1:49074)
0927 22:13:53.635016 (+ 14687us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:53.635042 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:53.635174 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:53.645994 (+ 10820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:53.645999 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:53.646029 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:53.646202 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:53.646211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:53.648356 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:53.648364 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:53.650255 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:53.650269 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:53.652039 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:53.652047 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:53.652230 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:53.652266 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:13:53.652366 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 22:13:53.652519 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14521,"thread_start_us":25206,"threads_started":1}
I20260927 22:13:53.653203 20442 heartbeater.cc:503] Master 127.0.97.62:38197 was elected leader, sending a full tablet report...
I20260927 22:13:53.654053 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:53.621062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49074 (local address 127.0.97.2:43587)
0927 22:13:53.644696 (+ 23634us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:53.644708 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:53.644748 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:13:53.644872 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:53.644883 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:53.645204 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:53.646423 (+ 1219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:53.646701 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:53.648195 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:53.650427 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:53.651824 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:53.652005 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:53.652227 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:53.652428 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 22:13:53.653719 (+ 1291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23395,"thread_start_us":24607,"threads_started":1}
I20260927 22:13:53.658442 20721 tablet_bootstrap.cc:492] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95: Bootstrap starting.
I20260927 22:13:53.657934 20508 tablet_service.cc:1467] Processing CreateTablet for tablet 3498bce12376493cb87d4bf60fbfa68d (DEFAULT_TABLE table=test-table [id=a90306dad6814f4f8a4b3e2193b46f45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:13:53.659696 20508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3498bce12376493cb87d4bf60fbfa68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:53.663096 20721 tablet_bootstrap.cc:654] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:53.664350 20721 log.cc:824] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:53.665518 20722 tablet_bootstrap.cc:492] T 3498bce12376493cb87d4bf60fbfa68d P c0e270745a0c4a4bbc2369f2d994753e: Bootstrap starting.
I20260927 22:13:53.668334 20721 tablet_bootstrap.cc:492] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95: No bootstrap required, opened a new log
I20260927 22:13:53.668644 20721 ts_tablet_manager.cc:1397] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 22:13:53.670341 20722 tablet_bootstrap.cc:654] T 3498bce12376493cb87d4bf60fbfa68d P c0e270745a0c4a4bbc2369f2d994753e: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:53.671509 20724 tablet_bootstrap.cc:492] T 3498bce12376493cb87d4bf60fbfa68d P 63fca6ed497542dc864e980c7194474a: Bootstrap starting.
I20260927 22:13:53.671594 20722 log.cc:824] T 3498bce12376493cb87d4bf60fbfa68d P c0e270745a0c4a4bbc2369f2d994753e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:53.674479 20724 tablet_bootstrap.cc:654] T 3498bce12376493cb87d4bf60fbfa68d P 63fca6ed497542dc864e980c7194474a: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:53.676052 20724 log.cc:824] T 3498bce12376493cb87d4bf60fbfa68d P 63fca6ed497542dc864e980c7194474a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:53.677382 20722 tablet_bootstrap.cc:492] T 3498bce12376493cb87d4bf60fbfa68d P c0e270745a0c4a4bbc2369f2d994753e: No bootstrap required, opened a new log
I20260927 22:13:53.677677 20722 ts_tablet_manager.cc:1397] T 3498bce12376493cb87d4bf60fbfa68d P c0e270745a0c4a4bbc2369f2d994753e: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260927 22:13:53.679095 20724 tablet_bootstrap.cc:492] T 3498bce12376493cb87d4bf60fbfa68d P 63fca6ed497542dc864e980c7194474a: No bootstrap required, opened a new log
I20260927 22:13:53.679353 20724 ts_tablet_manager.cc:1397] T 3498bce12376493cb87d4bf60fbfa68d P 63fca6ed497542dc864e980c7194474a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 22:13:53.679505 20721 raft_consensus.cc:348] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da1069de647b4c538b111246e441df95" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42857 } } peers { permanent_uuid: "63fca6ed497542dc864e980c7194474a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "c0e270745a0c4a4bbc2369f2d994753e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37319 } }
I20260927 22:13:53.679886 20721 raft_consensus.cc:374] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:53.680052 20721 raft_consensus.cc:729] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da1069de647b4c538b111246e441df95, State: Initialized, Role: FOLLOWER
I20260927 22:13:53.680653 20721 consensus_queue.cc:260] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da1069de647b4c538b111246e441df95" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42857 } } peers { permanent_uuid: "63fca6ed497542dc864e980c7194474a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "c0e270745a0c4a4bbc2369f2d994753e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37319 } }
I20260927 22:13:53.682400 20721 ts_tablet_manager.cc:1428] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260927 22:13:53.688802 20722 raft_consensus.cc:348] T 3498bce12376493cb87d4bf60fbfa68d P c0e270745a0c4a4bbc2369f2d994753e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da1069de647b4c538b111246e441df95" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42857 } } peers { permanent_uuid: "63fca6ed497542dc864e980c7194474a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "c0e270745a0c4a4bbc2369f2d994753e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37319 } }
I20260927 22:13:53.689270 20722 raft_consensus.cc:374] T 3498bce12376493cb87d4bf60fbfa68d P c0e270745a0c4a4bbc2369f2d994753e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:53.689375 20722 raft_consensus.cc:729] T 3498bce12376493cb87d4bf60fbfa68d P c0e270745a0c4a4bbc2369f2d994753e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0e270745a0c4a4bbc2369f2d994753e, State: Initialized, Role: FOLLOWER
I20260927 22:13:53.689800 20724 raft_consensus.cc:348] T 3498bce12376493cb87d4bf60fbfa68d P 63fca6ed497542dc864e980c7194474a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da1069de647b4c538b111246e441df95" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42857 } } peers { permanent_uuid: "63fca6ed497542dc864e980c7194474a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "c0e270745a0c4a4bbc2369f2d994753e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37319 } }
I20260927 22:13:53.690024 20722 consensus_queue.cc:260] T 3498bce12376493cb87d4bf60fbfa68d P c0e270745a0c4a4bbc2369f2d994753e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da1069de647b4c538b111246e441df95" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42857 } } peers { permanent_uuid: "63fca6ed497542dc864e980c7194474a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "c0e270745a0c4a4bbc2369f2d994753e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37319 } }
I20260927 22:13:53.690253 20724 raft_consensus.cc:374] T 3498bce12376493cb87d4bf60fbfa68d P 63fca6ed497542dc864e980c7194474a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:53.690354 20724 raft_consensus.cc:729] T 3498bce12376493cb87d4bf60fbfa68d P 63fca6ed497542dc864e980c7194474a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63fca6ed497542dc864e980c7194474a, State: Initialized, Role: FOLLOWER
I20260927 22:13:53.691188 20703 heartbeater.cc:503] Master 127.0.97.62:38197 was elected leader, sending a full tablet report...
I20260927 22:13:53.691887 20722 ts_tablet_manager.cc:1428] T 3498bce12376493cb87d4bf60fbfa68d P c0e270745a0c4a4bbc2369f2d994753e: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260927 22:13:53.692147 20724 consensus_queue.cc:260] T 3498bce12376493cb87d4bf60fbfa68d P 63fca6ed497542dc864e980c7194474a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da1069de647b4c538b111246e441df95" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42857 } } peers { permanent_uuid: "63fca6ed497542dc864e980c7194474a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "c0e270745a0c4a4bbc2369f2d994753e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37319 } }
I20260927 22:13:53.693400 20573 heartbeater.cc:503] Master 127.0.97.62:38197 was elected leader, sending a full tablet report...
I20260927 22:13:53.693828 20724 ts_tablet_manager.cc:1428] T 3498bce12376493cb87d4bf60fbfa68d P 63fca6ed497542dc864e980c7194474a: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260927 22:13:53.811408 20704 tablet.cc:2369] T 3498bce12376493cb87d4bf60fbfa68d P c0e270745a0c4a4bbc2369f2d994753e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:13:53.846993 20574 tablet.cc:2369] T 3498bce12376493cb87d4bf60fbfa68d P 63fca6ed497542dc864e980c7194474a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:13:53.879446 20443 tablet.cc:2369] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:13:54.080693 20727 raft_consensus.cc:482] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:13:54.081004 20727 raft_consensus.cc:504] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da1069de647b4c538b111246e441df95" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42857 } } peers { permanent_uuid: "63fca6ed497542dc864e980c7194474a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "c0e270745a0c4a4bbc2369f2d994753e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37319 } }
I20260927 22:13:54.082815 20727 leader_election.cc:288] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63fca6ed497542dc864e980c7194474a (127.0.97.2:43587), c0e270745a0c4a4bbc2369f2d994753e (127.0.97.3:37319)
I20260927 22:13:54.090095 20720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:54.083236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54787 (local address 127.0.97.3:37319)
0927 22:13:54.083435 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:54.083441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:54.084548 (+ 1107us) server_negotiation.cc:407] Connection header received
0927 22:13:54.084689 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:54.084693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:54.084785 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:54.084882 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:54.085493 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:54.086205 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:54.088402 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:54.089182 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:54.089316 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:54.089425 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:54.089715 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 22:13:54.089838 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:13:54.090107 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:54.083236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56351 (local address 127.0.97.2:43587)
0927 22:13:54.083454 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:54.083461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:54.084405 (+ 944us) server_negotiation.cc:407] Connection header received
0927 22:13:54.084560 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:54.084566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:54.084669 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:54.084748 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:13:54.085402 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:54.086130 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:54.088290 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:54.089186 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:54.089336 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:54.089435 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:54.089738 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 22:13:54.089861 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 22:13:54.090212 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:54.083480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37319 (local address 127.0.97.1:54787)
0927 22:13:54.084395 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:54.084415 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:54.084563 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:54.084914 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:54.084917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:54.084947 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:54.085297 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:54.085304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:54.086349 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:54.086352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:54.088221 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:54.088238 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:54.089367 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:54.089373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:54.089552 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:54.089583 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:13:54.089656 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 22:13:54.089806 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":74,"thread_start_us":452,"threads_started":1}
I20260927 22:13:54.090236 20730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:54.083412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43587 (local address 127.0.97.1:56351)
0927 22:13:54.084266 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0927 22:13:54.084287 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:13:54.084422 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:13:54.084789 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:13:54.084793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:13:54.084819 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:13:54.085249 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:54.085261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:54.086263 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:54.086270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:54.088084 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:13:54.088096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:13:54.089403 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:13:54.089410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:13:54.089563 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:54.089591 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 22:13:54.089698 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 22:13:54.089823 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":76,"thread_start_us":484,"threads_started":1}
I20260927 22:13:54.091317 20658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3498bce12376493cb87d4bf60fbfa68d" candidate_uuid: "da1069de647b4c538b111246e441df95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0e270745a0c4a4bbc2369f2d994753e" is_pre_election: true
I20260927 22:13:54.091470 20528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3498bce12376493cb87d4bf60fbfa68d" candidate_uuid: "da1069de647b4c538b111246e441df95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63fca6ed497542dc864e980c7194474a" is_pre_election: true
I20260927 22:13:54.091813 20658 raft_consensus.cc:2454] T 3498bce12376493cb87d4bf60fbfa68d P c0e270745a0c4a4bbc2369f2d994753e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da1069de647b4c538b111246e441df95 in term 0.
I20260927 22:13:54.091988 20528 raft_consensus.cc:2454] T 3498bce12376493cb87d4bf60fbfa68d P 63fca6ed497542dc864e980c7194474a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da1069de647b4c538b111246e441df95 in term 0.
I20260927 22:13:54.092480 20330 leader_election.cc:302] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0e270745a0c4a4bbc2369f2d994753e, da1069de647b4c538b111246e441df95; no voters:
I20260927 22:13:54.092950 20727 raft_consensus.cc:2781] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:13:54.093160 20727 raft_consensus.cc:482] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:13:54.093293 20727 raft_consensus.cc:3037] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:54.095561 20727 raft_consensus.cc:504] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da1069de647b4c538b111246e441df95" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42857 } } peers { permanent_uuid: "63fca6ed497542dc864e980c7194474a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "c0e270745a0c4a4bbc2369f2d994753e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37319 } }
I20260927 22:13:54.096565 20727 leader_election.cc:288] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95 [CANDIDATE]: Term 1 election: Requested vote from peers 63fca6ed497542dc864e980c7194474a (127.0.97.2:43587), c0e270745a0c4a4bbc2369f2d994753e (127.0.97.3:37319)
I20260927 22:13:54.096771 20658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3498bce12376493cb87d4bf60fbfa68d" candidate_uuid: "da1069de647b4c538b111246e441df95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0e270745a0c4a4bbc2369f2d994753e"
I20260927 22:13:54.096812 20528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3498bce12376493cb87d4bf60fbfa68d" candidate_uuid: "da1069de647b4c538b111246e441df95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63fca6ed497542dc864e980c7194474a"
I20260927 22:13:54.097014 20528 raft_consensus.cc:3037] T 3498bce12376493cb87d4bf60fbfa68d P 63fca6ed497542dc864e980c7194474a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:54.096984 20658 raft_consensus.cc:3037] T 3498bce12376493cb87d4bf60fbfa68d P c0e270745a0c4a4bbc2369f2d994753e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:54.100153 20658 raft_consensus.cc:2454] T 3498bce12376493cb87d4bf60fbfa68d P c0e270745a0c4a4bbc2369f2d994753e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da1069de647b4c538b111246e441df95 in term 1.
I20260927 22:13:54.100273 20528 raft_consensus.cc:2454] T 3498bce12376493cb87d4bf60fbfa68d P 63fca6ed497542dc864e980c7194474a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da1069de647b4c538b111246e441df95 in term 1.
I20260927 22:13:54.100610 20330 leader_election.cc:302] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0e270745a0c4a4bbc2369f2d994753e, da1069de647b4c538b111246e441df95; no voters:
I20260927 22:13:54.101079 20727 raft_consensus.cc:2781] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:13:54.101619 20727 raft_consensus.cc:686] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95 [term 1 LEADER]: Becoming Leader. State: Replica: da1069de647b4c538b111246e441df95, State: Running, Role: LEADER
I20260927 22:13:54.102133 20727 consensus_queue.cc:237] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da1069de647b4c538b111246e441df95" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42857 } } peers { permanent_uuid: "63fca6ed497542dc864e980c7194474a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } } peers { permanent_uuid: "c0e270745a0c4a4bbc2369f2d994753e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37319 } }
I20260927 22:13:54.106751 20255 catalog_manager.cc:5495] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95 reported cstate change: term changed from 0 to 1, leader changed from <none> to da1069de647b4c538b111246e441df95 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "da1069de647b4c538b111246e441df95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da1069de647b4c538b111246e441df95" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63fca6ed497542dc864e980c7194474a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0e270745a0c4a4bbc2369f2d994753e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37319 } health_report { overall_health: UNKNOWN } } }
I20260927 22:13:54.222441 20736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:54.216320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57388 (local address 127.0.97.1:42857)
0927 22:13:54.216891 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:54.216898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:54.217036 (+ 138us) server_negotiation.cc:407] Connection header received
0927 22:13:54.217294 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:54.217298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:54.217392 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:54.217564 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:13:54.218003 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:54.218750 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:54.220268 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:54.220681 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:54.220818 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:54.221296 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:13:54.221806 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:13:54.221962 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:54.222079 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 22:13:54.222160 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":341,"threads_started":1}
I20260927 22:13:54.230479 20528 raft_consensus.cc:1261] T 3498bce12376493cb87d4bf60fbfa68d P 63fca6ed497542dc864e980c7194474a [term 1 FOLLOWER]: Refusing update from remote peer da1069de647b4c538b111246e441df95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:13:54.230489 20658 raft_consensus.cc:1261] T 3498bce12376493cb87d4bf60fbfa68d P c0e270745a0c4a4bbc2369f2d994753e [term 1 FOLLOWER]: Refusing update from remote peer da1069de647b4c538b111246e441df95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:13:54.231575 20732 consensus_queue.cc:1035] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63fca6ed497542dc864e980c7194474a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:13:54.231913 20727 consensus_queue.cc:1035] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0e270745a0c4a4bbc2369f2d994753e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:13:54.246912 20737 mvcc.cc:205] Tried to move back new op lower bound from 7334081471397277696 to 7334081470886699008. Current Snapshot: MvccSnapshot[applied={T|T < 7334081471397277696}]
I20260927 22:13:54.261312 20255 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:13:54.261606 20255 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:13:54.264856 20255 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c089c5f81f141d6b0fce05ff2700f15: Sending DeleteTablet for 3 replicas of tablet 3498bce12376493cb87d4bf60fbfa68d
I20260927 22:13:54.266034 20508 tablet_service.cc:1514] Processing DeleteTablet for tablet 3498bce12376493cb87d4bf60fbfa68d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:13:54 UTC) from {username='slave'} at 127.0.0.1:49074
I20260927 22:13:54.266244 20377 tablet_service.cc:1514] Processing DeleteTablet for tablet 3498bce12376493cb87d4bf60fbfa68d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:13:54 UTC) from {username='slave'} at 127.0.0.1:57382
I20260927 22:13:54.266284 20638 tablet_service.cc:1514] Processing DeleteTablet for tablet 3498bce12376493cb87d4bf60fbfa68d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:13:54 UTC) from {username='slave'} at 127.0.0.1:43098
I20260927 22:13:54.267990 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 20315
I20260927 22:13:54.278371 20748 tablet_replica.cc:330] T 3498bce12376493cb87d4bf60fbfa68d P 63fca6ed497542dc864e980c7194474a: stopping tablet replica
I20260927 22:13:54.279012 20748 raft_consensus.cc:2229] T 3498bce12376493cb87d4bf60fbfa68d P 63fca6ed497542dc864e980c7194474a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:13:54.279616 20748 raft_consensus.cc:2258] T 3498bce12376493cb87d4bf60fbfa68d P 63fca6ed497542dc864e980c7194474a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:13:54.281677 20748 ts_tablet_manager.cc:1905] T 3498bce12376493cb87d4bf60fbfa68d P 63fca6ed497542dc864e980c7194474a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:13:54.288933 20748 ts_tablet_manager.cc:1918] T 3498bce12376493cb87d4bf60fbfa68d P 63fca6ed497542dc864e980c7194474a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:13:54.289165 20748 log.cc:1196] T 3498bce12376493cb87d4bf60fbfa68d P 63fca6ed497542dc864e980c7194474a: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-1/wal/wals/3498bce12376493cb87d4bf60fbfa68d
I20260927 22:13:54.289688 20748 ts_tablet_manager.cc:1939] T 3498bce12376493cb87d4bf60fbfa68d P 63fca6ed497542dc864e980c7194474a: Deleting consensus metadata
I20260927 22:13:54.291782 20241 catalog_manager.cc:4841] TS 63fca6ed497542dc864e980c7194474a (127.0.97.2:43587): tablet 3498bce12376493cb87d4bf60fbfa68d (table test-table [id=a90306dad6814f4f8a4b3e2193b46f45]) successfully deleted
I20260927 22:13:54.293979 20750 tablet_replica.cc:330] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95: stopping tablet replica
I20260927 22:13:54.294708 20749 tablet_replica.cc:330] T 3498bce12376493cb87d4bf60fbfa68d P c0e270745a0c4a4bbc2369f2d994753e: stopping tablet replica
I20260927 22:13:54.295179 20750 raft_consensus.cc:2229] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:13:54.295207 20749 raft_consensus.cc:2229] T 3498bce12376493cb87d4bf60fbfa68d P c0e270745a0c4a4bbc2369f2d994753e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:13:54.295773 20749 raft_consensus.cc:2258] T 3498bce12376493cb87d4bf60fbfa68d P c0e270745a0c4a4bbc2369f2d994753e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:13:54.295895 20750 raft_consensus.cc:2258] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:13:54.297992 20749 ts_tablet_manager.cc:1905] T 3498bce12376493cb87d4bf60fbfa68d P c0e270745a0c4a4bbc2369f2d994753e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:13:54.298019 20750 ts_tablet_manager.cc:1905] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:13:54.305363 20749 ts_tablet_manager.cc:1918] T 3498bce12376493cb87d4bf60fbfa68d P c0e270745a0c4a4bbc2369f2d994753e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:13:54.305645 20749 log.cc:1196] T 3498bce12376493cb87d4bf60fbfa68d P c0e270745a0c4a4bbc2369f2d994753e: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-2/wal/wals/3498bce12376493cb87d4bf60fbfa68d
I20260927 22:13:54.305639 20750 ts_tablet_manager.cc:1918] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:13:54.305800 20750 log.cc:1196] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95: Deleting WAL directory at /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546912983837-388-0/minicluster-data/ts-0/wal/wals/3498bce12376493cb87d4bf60fbfa68d
I20260927 22:13:54.306264 20749 ts_tablet_manager.cc:1939] T 3498bce12376493cb87d4bf60fbfa68d P c0e270745a0c4a4bbc2369f2d994753e: Deleting consensus metadata
I20260927 22:13:54.306350 20750 ts_tablet_manager.cc:1939] T 3498bce12376493cb87d4bf60fbfa68d P da1069de647b4c538b111246e441df95: Deleting consensus metadata
I20260927 22:13:54.306821 20736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:54.270988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57404 (local address 127.0.97.1:42857)
0927 22:13:54.271157 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:54.271165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:54.292034 (+ 20869us) server_negotiation.cc:407] Connection header received
0927 22:13:54.292642 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:54.292648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:54.292772 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:54.292893 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:54.293612 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:54.294899 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:54.296818 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:54.297499 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:54.297597 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:54.304901 (+ 7304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:54.304915 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:54.304939 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:54.305802 (+ 863us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:54.305872 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:54.305897 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:54.305988 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:54.306259 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 22:13:54.306395 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 22:13:54.308180 20437 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:54.308314 20240 catalog_manager.cc:4841] TS da1069de647b4c538b111246e441df95 (127.0.97.1:42857): tablet 3498bce12376493cb87d4bf60fbfa68d (table test-table [id=a90306dad6814f4f8a4b3e2193b46f45]) successfully deleted
I20260927 22:13:54.309052 20240 catalog_manager.cc:4841] TS c0e270745a0c4a4bbc2369f2d994753e (127.0.97.3:37319): tablet 3498bce12376493cb87d4bf60fbfa68d (table test-table [id=a90306dad6814f4f8a4b3e2193b46f45]) successfully deleted
I20260927 22:13:54.409070 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 20315
I20260927 22:13:54.428218 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 20446
I20260927 22:13:54.435356 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:54.429019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49078 (local address 127.0.97.2:43587)
0927 22:13:54.429227 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:54.429235 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:54.429255 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:13:54.429461 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:54.429472 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:54.429579 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:54.429692 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:54.430064 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:54.430969 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:54.432229 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:54.432639 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:54.432767 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:54.433845 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:54.433854 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:54.433870 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:54.434479 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:54.434535 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:54.434556 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:54.434776 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:54.434907 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 22:13:54.435023 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:13:54.436115 20568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:54.528123 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 20446
I20260927 22:13:54.545477 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 20576
I20260927 22:13:54.552449 20720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:54.546432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43110 (local address 127.0.97.3:37319)
0927 22:13:54.546875 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:54.546880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:54.546908 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:13:54.546975 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:54.546981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:54.547101 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:54.547206 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:54.547576 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:54.548326 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:54.549550 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:54.549953 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:54.550047 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:54.551092 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:54.551101 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:54.551116 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:54.551718 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:54.551773 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:54.551800 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:54.551891 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:54.551995 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 22:13:54.552082 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 22:13:54.553237 20698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:54.643491 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 20576
I20260927 22:13:54.660346 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 20225
I20260927 22:13:54.661558 20286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:54.771209 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 20225
2026-09-27T22:13:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3204 ms)
[----------] 6 tests from EncryptionPolicyTest (22360 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:13:54 dist-test-slave-mp03 krb5kdc[20758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:13:54 dist-test-slave-mp03 krb5kdc[20758](info): set up 2 sockets
Sep 27 22:13:54 dist-test-slave-mp03 krb5kdc[20758](info): commencing operation
krb5kdc: starting...
W20260927 22:13:55.911293 388 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:13:56 dist-test-slave-mp03 krb5kdc[20758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:13:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:13:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:13:56.176041 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38827
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:13:56.509079 20771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:56.509450 20771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:56.509665 20771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:56.517021 20771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:13:56.517186 20771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:56.517314 20771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:56.517426 20771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:13:56.517511 20771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:13:56.527406 20771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38827
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:56.528797 20771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:56.530113 20771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:56.542268 20779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:56.542296 20776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:56.542356 20777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:56.543130 20771 server_base.cc:1034] running on GCE node
I20260927 22:13:56.543964 20771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:56.545400 20771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:56.546691 20771 hybrid_clock.cc:648] HybridClock initialized: now 1790547236546641 us; error 43 us; skew 500 ppm
I20260927 22:13:56.547104 20771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:13:56 dist-test-slave-mp03 krb5kdc[20758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:13:56.550547 20771 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:13:56.552208 20771 webserver.cc:460] Webserver started at http://127.0.0.1:42877/ using document root <none> and password file <none>
I20260927 22:13:56.552932 20771 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:56.553090 20771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:56.553403 20771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:56.556548 20771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "a0cf97124198408db5362a4577a0e339"
format_stamp: "Formatted at 2026-09-27 22:13:56 on dist-test-slave-mp03"
server_key: "47bb5928747cd0468b85fb974b6604e0"
server_key_iv: "448034bf2fcad66031adb5736b969325"
server_key_version: "encryptionkey@0"
I20260927 22:13:56.557318 20771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "a0cf97124198408db5362a4577a0e339"
format_stamp: "Formatted at 2026-09-27 22:13:56 on dist-test-slave-mp03"
server_key: "47bb5928747cd0468b85fb974b6604e0"
server_key_iv: "448034bf2fcad66031adb5736b969325"
server_key_version: "encryptionkey@0"
I20260927 22:13:56.562958 20771 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:13:56.566836 20786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:56.568287 20771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:13:56.568504 20771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "a0cf97124198408db5362a4577a0e339"
format_stamp: "Formatted at 2026-09-27 22:13:56 on dist-test-slave-mp03"
server_key: "47bb5928747cd0468b85fb974b6604e0"
server_key_iv: "448034bf2fcad66031adb5736b969325"
server_key_version: "encryptionkey@0"
I20260927 22:13:56.568783 20771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:56.592231 20771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:56.596509 20771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:56.596824 20771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:56.615818 20771 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38827
I20260927 22:13:56.615861 20837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38827 every 8 connection(s)
I20260927 22:13:56.617304 20771 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:13:56.620662 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 20771
I20260927 22:13:56.620885 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:13:56.621420 20838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:56.621726 388 external_mini_cluster.cc:1428] Setting key 6d9173025e56fa6ca1afd1bd614c2eca
I20260927 22:13:56.636335 20838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0cf97124198408db5362a4577a0e339: Bootstrap starting.
Sep 27 22:13:56 dist-test-slave-mp03 krb5kdc[20758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:13:56.639823 20838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0cf97124198408db5362a4577a0e339: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:56.640957 20838 log.cc:824] T 00000000000000000000000000000000 P a0cf97124198408db5362a4577a0e339: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:56.643985 20838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0cf97124198408db5362a4577a0e339: No bootstrap required, opened a new log
I20260927 22:13:56.645622 20841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:56.623664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52807 (local address 127.0.0.1:38827)
0927 22:13:56.624435 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:56.624445 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:56.624488 (+ 43us) server_negotiation.cc:407] Connection header received
0927 22:13:56.626309 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:56.626330 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:56.626817 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:56.627151 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:56.628938 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:56.630254 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:56.632130 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:56.632871 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:56.633018 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:56.637992 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:56.638000 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:56.638015 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:56.638065 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:56.641398 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:56.642230 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:56.642237 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:56.642244 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:56.642410 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:56.642956 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:56.642960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:56.642962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:56.643468 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:56.643793 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:56.644286 (+ 493us) server_negotiation.cc:299] Negotiation successful
0927 22:13:56.644620 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":370,"threads_started":1}
I20260927 22:13:56.649797 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:13:56.652352 20838 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0cf97124198408db5362a4577a0e339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0cf97124198408db5362a4577a0e339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38827 } }
I20260927 22:13:56.652802 20838 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0cf97124198408db5362a4577a0e339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:56.652922 20838 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0cf97124198408db5362a4577a0e339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0cf97124198408db5362a4577a0e339, State: Initialized, Role: FOLLOWER
I20260927 22:13:56.653613 20838 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0cf97124198408db5362a4577a0e339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0cf97124198408db5362a4577a0e339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38827 } }
I20260927 22:13:56.653867 20838 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0cf97124198408db5362a4577a0e339 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:13:56.654003 20838 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0cf97124198408db5362a4577a0e339 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:13:56.654174 20838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0cf97124198408db5362a4577a0e339 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:56.656924 20838 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0cf97124198408db5362a4577a0e339 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0cf97124198408db5362a4577a0e339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38827 } }
I20260927 22:13:56.657559 20838 leader_election.cc:302] T 00000000000000000000000000000000 P a0cf97124198408db5362a4577a0e339 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0cf97124198408db5362a4577a0e339; no voters:
I20260927 22:13:56.658264 20838 leader_election.cc:288] T 00000000000000000000000000000000 P a0cf97124198408db5362a4577a0e339 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:13:56.658384 20848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0cf97124198408db5362a4577a0e339 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:13:56.659684 20848 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0cf97124198408db5362a4577a0e339 [term 1 LEADER]: Becoming Leader. State: Replica: a0cf97124198408db5362a4577a0e339, State: Running, Role: LEADER
I20260927 22:13:56.660305 20848 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0cf97124198408db5362a4577a0e339 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0cf97124198408db5362a4577a0e339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38827 } }
I20260927 22:13:56.660679 20838 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0cf97124198408db5362a4577a0e339 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:13:56.663350 20850 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0cf97124198408db5362a4577a0e339 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0cf97124198408db5362a4577a0e339. Latest consensus state: current_term: 1 leader_uuid: "a0cf97124198408db5362a4577a0e339" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0cf97124198408db5362a4577a0e339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38827 } } }
I20260927 22:13:56.663740 20850 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0cf97124198408db5362a4577a0e339 [sys.catalog]: This master's current role is: LEADER
I20260927 22:13:56.664279 20849 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0cf97124198408db5362a4577a0e339 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0cf97124198408db5362a4577a0e339" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0cf97124198408db5362a4577a0e339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38827 } } }
I20260927 22:13:56.664587 20849 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0cf97124198408db5362a4577a0e339 [sys.catalog]: This master's current role is: LEADER
I20260927 22:13:56.675937 20841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:56.654789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44913 (local address 127.0.0.1:38827)
0927 22:13:56.655006 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:56.655013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:56.655470 (+ 457us) server_negotiation.cc:407] Connection header received
0927 22:13:56.655613 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:56.655619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:56.655712 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:56.655825 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:56.657394 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:56.658655 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:56.660478 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:56.661196 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:56.661456 (+ 260us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:56.665333 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:56.665344 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:56.665352 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:13:56.665407 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:56.670197 (+ 4790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:56.671506 (+ 1309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:56.671514 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:56.671520 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:56.671706 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:13:56.672515 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:13:56.672523 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:13:56.672529 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:13:56.673134 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:56.673560 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:56.675123 (+ 1563us) server_negotiation.cc:299] Negotiation successful
0927 22:13:56.675379 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":69}
I20260927 22:13:56.679986 20853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:13:56.688941 20853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:13:56.701202 20853 catalog_manager.cc:1347] Generated new cluster ID: e1a3ad0c11284367b2becaae1c5d737f
I20260927 22:13:56.701344 20853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:13:56.721822 20853 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:13:56.723850 20853 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:13:56.735261 20853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0cf97124198408db5362a4577a0e339: Generated new TSK 0
I20260927 22:13:56.736423 20853 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:13:56.742977 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 20771
I20260927 22:13:56.744800 20833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:13:56.835772 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 20771
W20260927 22:13:56.854446 20756 connection.cc:535] client connection to 127.0.0.1:38827 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38827): Connection reset by peer (error 104)
2026-09-27T22:13:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2059 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:13:56 dist-test-slave-mp03 krb5kdc[20871](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:13:56 dist-test-slave-mp03 krb5kdc[20871](info): set up 2 sockets
Sep 27 22:13:56 dist-test-slave-mp03 krb5kdc[20871](info): commencing operation
krb5kdc: starting...
W20260927 22:13:57.966943 388 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:13:58 dist-test-slave-mp03 krb5kdc[20871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:13:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:13:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:13:58.230818 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41603
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:13:58.566149 20884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:13:58.566517 20884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:13:58.566699 20884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:13:58.574191 20884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:13:58.574357 20884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:13:58.574458 20884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:13:58.574579 20884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:13:58.574687 20884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:13:58.584933 20884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41603
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:13:58.586304 20884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:13:58.587760 20884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:13:58.599939 20889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:58.599905 20890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:13:58.599984 20892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:13:58.600641 20884 server_base.cc:1034] running on GCE node
I20260927 22:13:58.601545 20884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:13:58.603235 20884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:13:58.604472 20884 hybrid_clock.cc:648] HybridClock initialized: now 1790547238604408 us; error 60 us; skew 500 ppm
I20260927 22:13:58.604954 20884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:13:58 dist-test-slave-mp03 krb5kdc[20871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:13:58.608553 20884 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:13:58.610056 20884 webserver.cc:460] Webserver started at http://127.0.0.1:42911/ using document root <none> and password file <none>
I20260927 22:13:58.610826 20884 fs_manager.cc:359] Metadata directory not provided
I20260927 22:13:58.610976 20884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:13:58.611282 20884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:13:58.614367 20884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "820d564a77774b9f9e417973067949dc"
format_stamp: "Formatted at 2026-09-27 22:13:58 on dist-test-slave-mp03"
server_key: "f9ebb0590ae63c717b807a56e2f2ffba"
server_key_iv: "7d8aac7f41bd6756d0232137e02dbc91"
server_key_version: "encryptionkey@0"
I20260927 22:13:58.615192 20884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "820d564a77774b9f9e417973067949dc"
format_stamp: "Formatted at 2026-09-27 22:13:58 on dist-test-slave-mp03"
server_key: "f9ebb0590ae63c717b807a56e2f2ffba"
server_key_iv: "7d8aac7f41bd6756d0232137e02dbc91"
server_key_version: "encryptionkey@0"
I20260927 22:13:58.620465 20884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:13:58.624130 20899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:13:58.625481 20884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:13:58.625663 20884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "820d564a77774b9f9e417973067949dc"
format_stamp: "Formatted at 2026-09-27 22:13:58 on dist-test-slave-mp03"
server_key: "f9ebb0590ae63c717b807a56e2f2ffba"
server_key_iv: "7d8aac7f41bd6756d0232137e02dbc91"
server_key_version: "encryptionkey@0"
I20260927 22:13:58.625870 20884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:13:58.638716 20884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:13:58.643189 20884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:13:58.643474 20884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:13:58.669255 20884 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41603
I20260927 22:13:58.669278 20950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41603 every 8 connection(s)
I20260927 22:13:58.670764 20884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:13:58.674401 20951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:13:58.675598 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 20884
I20260927 22:13:58.675782 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:13:58.676240 388 external_mini_cluster.cc:1428] Setting key d3c19a7320cc165b51aa507cc8d8d590
I20260927 22:13:58.688213 20951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 820d564a77774b9f9e417973067949dc: Bootstrap starting.
I20260927 22:13:58.689867 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:58.677919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55213 (local address 127.0.0.1:41603)
0927 22:13:58.678729 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:58.678740 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:58.678781 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:13:58.680667 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:58.680691 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:58.681095 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:58.681423 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:58.681988 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:58.683360 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:58.685321 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:58.686016 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:58.686143 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:58.687536 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:58.687543 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:58.687557 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:58.688101 (+ 544us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:58.688181 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:58.688209 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:58.688340 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:58.688657 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 22:13:58.688884 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":353,"threads_started":1}
I20260927 22:13:58.691814 20951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 820d564a77774b9f9e417973067949dc: Neither blocks nor log segments found. Creating new log.
I20260927 22:13:58.693022 20951 log.cc:824] T 00000000000000000000000000000000 P 820d564a77774b9f9e417973067949dc: Log is configured to *not* fsync() on all Append() calls
I20260927 22:13:58.693936 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:13:58.696427 20951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 820d564a77774b9f9e417973067949dc: No bootstrap required, opened a new log
I20260927 22:13:58.705919 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:13:58.698890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59213 (local address 127.0.0.1:41603)
0927 22:13:58.699071 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:13:58.699082 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:13:58.699609 (+ 527us) server_negotiation.cc:407] Connection header received
0927 22:13:58.699756 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:13:58.699762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:13:58.699840 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:13:58.699948 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:13:58.700397 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:58.701566 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:58.702932 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:13:58.703652 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:13:58.703772 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:13:58.704872 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:13:58.704878 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:13:58.704883 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:13:58.704904 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:13:58.704965 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:13:58.704990 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:13:58.705108 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 22:13:58.705293 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 22:13:58.705375 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:13:58.706718 20951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 820d564a77774b9f9e417973067949dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820d564a77774b9f9e417973067949dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41603 } }
I20260927 22:13:58.707226 20951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 820d564a77774b9f9e417973067949dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:13:58.707362 20951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 820d564a77774b9f9e417973067949dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820d564a77774b9f9e417973067949dc, State: Initialized, Role: FOLLOWER
I20260927 22:13:58.707958 20951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 820d564a77774b9f9e417973067949dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820d564a77774b9f9e417973067949dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41603 } }
I20260927 22:13:58.708186 20951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 820d564a77774b9f9e417973067949dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:13:58.708285 20951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 820d564a77774b9f9e417973067949dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:13:58.708403 20951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 820d564a77774b9f9e417973067949dc [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:13:58.710536 20951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 820d564a77774b9f9e417973067949dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820d564a77774b9f9e417973067949dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41603 } }
I20260927 22:13:58.711189 20951 leader_election.cc:302] T 00000000000000000000000000000000 P 820d564a77774b9f9e417973067949dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 820d564a77774b9f9e417973067949dc; no voters:
I20260927 22:13:58.711828 20951 leader_election.cc:288] T 00000000000000000000000000000000 P 820d564a77774b9f9e417973067949dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:13:58.711951 20961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 820d564a77774b9f9e417973067949dc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:13:58.712733 20961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 820d564a77774b9f9e417973067949dc [term 1 LEADER]: Becoming Leader. State: Replica: 820d564a77774b9f9e417973067949dc, State: Running, Role: LEADER
I20260927 22:13:58.713354 20961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 820d564a77774b9f9e417973067949dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820d564a77774b9f9e417973067949dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41603 } }
I20260927 22:13:58.714043 20951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 820d564a77774b9f9e417973067949dc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:13:58.716285 20962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 820d564a77774b9f9e417973067949dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "820d564a77774b9f9e417973067949dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820d564a77774b9f9e417973067949dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41603 } } }
I20260927 22:13:58.716259 20963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 820d564a77774b9f9e417973067949dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 820d564a77774b9f9e417973067949dc. Latest consensus state: current_term: 1 leader_uuid: "820d564a77774b9f9e417973067949dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820d564a77774b9f9e417973067949dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41603 } } }
I20260927 22:13:58.716715 20962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 820d564a77774b9f9e417973067949dc [sys.catalog]: This master's current role is: LEADER
I20260927 22:13:58.716739 20963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 820d564a77774b9f9e417973067949dc [sys.catalog]: This master's current role is: LEADER
I20260927 22:13:58.722337 20968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:13:58.726850 20968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:13:58.734937 20968 catalog_manager.cc:1347] Generated new cluster ID: 747f33a246574a2dab366851f81978a0
I20260927 22:13:58.735074 20968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:13:58.761498 20968 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:13:58.762956 20968 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:13:58.775285 20968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 820d564a77774b9f9e417973067949dc: Generated new TSK 0
I20260927 22:13:58.776219 20968 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:13:58.782153 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 20884
W20260927 22:13:58.784399 20946 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55213
W20260927 22:13:58.785148 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 20884: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:13:58.785337 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 20884
W20260927 22:13:58.799695 20869 connection.cc:535] client connection to 127.0.0.1:41603 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41603): Connection reset by peer (error 104)
2026-09-27T22:13:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1945 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:13:58 dist-test-slave-mp03 krb5kdc[20982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:13:58 dist-test-slave-mp03 krb5kdc[20982](info): set up 2 sockets
Sep 27 22:13:58 dist-test-slave-mp03 krb5kdc[20982](info): commencing operation
krb5kdc: starting...
W20260927 22:13:59.895020 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.008s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:14:00 dist-test-slave-mp03 krb5kdc[20982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:14:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:14:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:14:00.154215 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39363
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:00.490257 20995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:00.490561 20995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:00.490742 20995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:00.498867 20995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:00.498979 20995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:00.499060 20995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:00.499153 20995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:00.499212 20995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:00.509514 20995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39363
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:00.510948 20995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:00.512269 20995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:00.524348 21000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:00.524351 21001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:00.524349 21003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:00.525507 20995 server_base.cc:1034] running on GCE node
I20260927 22:14:00.526304 20995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:00.527757 20995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:00.528993 20995 hybrid_clock.cc:648] HybridClock initialized: now 1790547240528945 us; error 42 us; skew 500 ppm
I20260927 22:14:00.529369 20995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:00 dist-test-slave-mp03 krb5kdc[20982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:00.537073 20995 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:14:00.538630 20995 webserver.cc:460] Webserver started at http://127.0.0.1:36487/ using document root <none> and password file <none>
I20260927 22:14:00.539336 20995 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:00.539465 20995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:00.539799 20995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:00.542831 20995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "d5428000e16e40fb8de2110a3f993860"
format_stamp: "Formatted at 2026-09-27 22:14:00 on dist-test-slave-mp03"
server_key: "f8c19838849a102e9354dd0004f0598b"
server_key_iv: "e287ad3fe56abb07702243f7078a8ef1"
server_key_version: "encryptionkey@0"
I20260927 22:14:00.543607 20995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "d5428000e16e40fb8de2110a3f993860"
format_stamp: "Formatted at 2026-09-27 22:14:00 on dist-test-slave-mp03"
server_key: "f8c19838849a102e9354dd0004f0598b"
server_key_iv: "e287ad3fe56abb07702243f7078a8ef1"
server_key_version: "encryptionkey@0"
I20260927 22:14:00.548732 20995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:14:00.552110 21010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:00.553398 20995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:14:00.553589 20995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "d5428000e16e40fb8de2110a3f993860"
format_stamp: "Formatted at 2026-09-27 22:14:00 on dist-test-slave-mp03"
server_key: "f8c19838849a102e9354dd0004f0598b"
server_key_iv: "e287ad3fe56abb07702243f7078a8ef1"
server_key_version: "encryptionkey@0"
I20260927 22:14:00.553793 20995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:00.559708 20995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:00.563902 20995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:00.564204 20995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:00.582479 20995 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39363
I20260927 22:14:00.582515 21061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39363 every 8 connection(s)
I20260927 22:14:00.583906 20995 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:14:00.587512 21062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:00.589187 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 20995
I20260927 22:14:00.589372 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:14:00.589905 388 external_mini_cluster.cc:1428] Setting key d2ebb212aeb03a04b97ef72a2eda73a1
I20260927 22:14:00.597657 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:00.591749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37557 (local address 127.0.0.1:39363)
0927 22:14:00.592550 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:00.592562 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:00.592613 (+ 51us) server_negotiation.cc:407] Connection header received
0927 22:14:00.594271 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:00.594296 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:00.594755 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:00.595096 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:00.596118 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:00.596124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:00.596136 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:14:00.596150 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:00.596210 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:14:00.596228 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:00.596391 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:00.596511 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 22:14:00.596740 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":370,"threads_started":1}
I20260927 22:14:00.601176 21062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5428000e16e40fb8de2110a3f993860: Bootstrap starting.
I20260927 22:14:00.601671 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:14:00.604988 21062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5428000e16e40fb8de2110a3f993860: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:00.606369 21062 log.cc:824] T 00000000000000000000000000000000 P d5428000e16e40fb8de2110a3f993860: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:00.609623 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:00.607054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35411 (local address 127.0.0.1:39363)
0927 22:14:00.607293 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:00.607301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:00.607690 (+ 389us) server_negotiation.cc:407] Connection header received
0927 22:14:00.607882 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:00.607887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:00.607945 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:00.608057 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:00.608990 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:00.608996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:00.609000 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:14:00.609019 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:00.609071 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:14:00.609090 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:00.609213 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:00.609283 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 22:14:00.609340 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:14:00.610005 21062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5428000e16e40fb8de2110a3f993860: No bootstrap required, opened a new log
I20260927 22:14:00.617980 21062 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5428000e16e40fb8de2110a3f993860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5428000e16e40fb8de2110a3f993860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39363 } }
I20260927 22:14:00.618335 21062 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5428000e16e40fb8de2110a3f993860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:00.618405 21062 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5428000e16e40fb8de2110a3f993860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5428000e16e40fb8de2110a3f993860, State: Initialized, Role: FOLLOWER
I20260927 22:14:00.618997 21062 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5428000e16e40fb8de2110a3f993860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5428000e16e40fb8de2110a3f993860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39363 } }
I20260927 22:14:00.619212 21062 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5428000e16e40fb8de2110a3f993860 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:14:00.619310 21062 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5428000e16e40fb8de2110a3f993860 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:14:00.619434 21062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5428000e16e40fb8de2110a3f993860 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:00.621658 21062 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5428000e16e40fb8de2110a3f993860 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5428000e16e40fb8de2110a3f993860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39363 } }
I20260927 22:14:00.622243 21062 leader_election.cc:302] T 00000000000000000000000000000000 P d5428000e16e40fb8de2110a3f993860 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5428000e16e40fb8de2110a3f993860; no voters:
I20260927 22:14:00.622926 21062 leader_election.cc:288] T 00000000000000000000000000000000 P d5428000e16e40fb8de2110a3f993860 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:14:00.623075 21072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5428000e16e40fb8de2110a3f993860 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:00.623806 21072 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5428000e16e40fb8de2110a3f993860 [term 1 LEADER]: Becoming Leader. State: Replica: d5428000e16e40fb8de2110a3f993860, State: Running, Role: LEADER
I20260927 22:14:00.624336 21072 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5428000e16e40fb8de2110a3f993860 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5428000e16e40fb8de2110a3f993860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39363 } }
I20260927 22:14:00.625263 21062 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5428000e16e40fb8de2110a3f993860 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:14:00.627272 21073 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5428000e16e40fb8de2110a3f993860 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5428000e16e40fb8de2110a3f993860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5428000e16e40fb8de2110a3f993860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39363 } } }
I20260927 22:14:00.627775 21073 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5428000e16e40fb8de2110a3f993860 [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:00.628561 21074 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5428000e16e40fb8de2110a3f993860 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5428000e16e40fb8de2110a3f993860. Latest consensus state: current_term: 1 leader_uuid: "d5428000e16e40fb8de2110a3f993860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5428000e16e40fb8de2110a3f993860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39363 } } }
I20260927 22:14:00.628907 21074 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5428000e16e40fb8de2110a3f993860 [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:00.630244 21079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:14:00.638047 21079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:14:00.647330 21079 catalog_manager.cc:1347] Generated new cluster ID: 1cd4d40c9d8e446f921f870f0f6f0d0e
I20260927 22:14:00.647449 21079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:14:00.675037 21079 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:14:00.675559 21079 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:14:00.695156 21079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5428000e16e40fb8de2110a3f993860: Generated new TSK 0
I20260927 22:14:00.696231 21079 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:14:00.709386 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 20995
W20260927 22:14:00.710609 21057 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37557
W20260927 22:14:00.711454 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 20995: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:14:00.711591 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 20995
2026-09-27T22:14:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1926 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:14:00 dist-test-slave-mp03 krb5kdc[21093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:14:00 dist-test-slave-mp03 krb5kdc[21093](info): set up 2 sockets
Sep 27 22:14:00 dist-test-slave-mp03 krb5kdc[21093](info): commencing operation
krb5kdc: starting...
W20260927 22:14:01.819015 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:14:01 dist-test-slave-mp03 krb5kdc[21093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:14:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:14:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:14:02.074240 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38169
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:02.405236 21106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:02.405536 21106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:02.405629 21106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:02.413251 21106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:02.413389 21106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:02.413439 21106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:02.413501 21106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:02.413546 21106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:02.423852 21106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:02.425158 21106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:02.426414 21106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:02.442852 21114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:02.442886 21112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:02.442955 21111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:02.444087 21106 server_base.cc:1034] running on GCE node
I20260927 22:14:02.444960 21106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:02.446468 21106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:02.447744 21106 hybrid_clock.cc:648] HybridClock initialized: now 1790547242447696 us; error 42 us; skew 500 ppm
I20260927 22:14:02.448185 21106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:02 dist-test-slave-mp03 krb5kdc[21093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:02.451517 21106 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:14:02.453378 21106 webserver.cc:460] Webserver started at http://127.0.0.1:42917/ using document root <none> and password file <none>
I20260927 22:14:02.454082 21106 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:02.454196 21106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:02.454468 21106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:02.457748 21106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "7d8fd1ea29c44fa786a346f878aec471"
format_stamp: "Formatted at 2026-09-27 22:14:02 on dist-test-slave-mp03"
server_key: "ff2b8f39efd6359679da671df0e942d8"
server_key_iv: "113f0daffcda6f89858bd1bc360d8dbd"
server_key_version: "encryptionkey@0"
I20260927 22:14:02.458515 21106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "7d8fd1ea29c44fa786a346f878aec471"
format_stamp: "Formatted at 2026-09-27 22:14:02 on dist-test-slave-mp03"
server_key: "ff2b8f39efd6359679da671df0e942d8"
server_key_iv: "113f0daffcda6f89858bd1bc360d8dbd"
server_key_version: "encryptionkey@0"
I20260927 22:14:02.464028 21106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 22:14:02.467819 21121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:02.469254 21106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:14:02.469462 21106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "7d8fd1ea29c44fa786a346f878aec471"
format_stamp: "Formatted at 2026-09-27 22:14:02 on dist-test-slave-mp03"
server_key: "ff2b8f39efd6359679da671df0e942d8"
server_key_iv: "113f0daffcda6f89858bd1bc360d8dbd"
server_key_version: "encryptionkey@0"
I20260927 22:14:02.469671 21106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:02.496446 21106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:02.501261 21106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:02.501576 21106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:02.520457 21106 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38169
I20260927 22:14:02.520488 21172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38169 every 8 connection(s)
I20260927 22:14:02.521837 21106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:14:02.525578 21173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:02.528322 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 21106
I20260927 22:14:02.528523 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:14:02.529052 388 external_mini_cluster.cc:1428] Setting key d501a513c5fc1fbc53f04d37dac368f2
I20260927 22:14:02.538209 21173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d8fd1ea29c44fa786a346f878aec471: Bootstrap starting.
I20260927 22:14:02.542088 21173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d8fd1ea29c44fa786a346f878aec471: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:02.543514 21173 log.cc:824] T 00000000000000000000000000000000 P 7d8fd1ea29c44fa786a346f878aec471: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:14:02 dist-test-slave-mp03 krb5kdc[21093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:14:02.546756 21173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d8fd1ea29c44fa786a346f878aec471: No bootstrap required, opened a new log
I20260927 22:14:02.553122 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:02.531081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33903 (local address 127.0.0.1:38169)
0927 22:14:02.531931 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:02.531943 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:02.531984 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:14:02.533726 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:02.533751 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:02.534186 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:02.534552 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:02.536431 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:02.537821 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:02.539640 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:02.540399 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:02.540540 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:02.545701 (+ 5161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:02.545709 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:02.545724 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:02.545768 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:02.548899 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:02.549662 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:02.549668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:02.549674 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:02.549839 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:02.550227 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:02.550231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:02.550233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:02.550833 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:02.551170 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:02.551547 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 22:14:02.551981 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":411,"threads_started":1}
I20260927 22:14:02.554807 21173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d8fd1ea29c44fa786a346f878aec471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8fd1ea29c44fa786a346f878aec471" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38169 } }
I20260927 22:14:02.555190 21173 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d8fd1ea29c44fa786a346f878aec471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:02.555315 21173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d8fd1ea29c44fa786a346f878aec471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d8fd1ea29c44fa786a346f878aec471, State: Initialized, Role: FOLLOWER
I20260927 22:14:02.555904 21173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d8fd1ea29c44fa786a346f878aec471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8fd1ea29c44fa786a346f878aec471" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38169 } }
I20260927 22:14:02.556203 21173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d8fd1ea29c44fa786a346f878aec471 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:14:02.556362 21173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d8fd1ea29c44fa786a346f878aec471 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:14:02.556553 21173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d8fd1ea29c44fa786a346f878aec471 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:02.557092 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:14:02.558990 21173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d8fd1ea29c44fa786a346f878aec471 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8fd1ea29c44fa786a346f878aec471" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38169 } }
I20260927 22:14:02.559484 21173 leader_election.cc:302] T 00000000000000000000000000000000 P 7d8fd1ea29c44fa786a346f878aec471 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d8fd1ea29c44fa786a346f878aec471; no voters:
I20260927 22:14:02.560165 21173 leader_election.cc:288] T 00000000000000000000000000000000 P 7d8fd1ea29c44fa786a346f878aec471 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:14:02.560305 21180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d8fd1ea29c44fa786a346f878aec471 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:02.562901 21180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d8fd1ea29c44fa786a346f878aec471 [term 1 LEADER]: Becoming Leader. State: Replica: 7d8fd1ea29c44fa786a346f878aec471, State: Running, Role: LEADER
I20260927 22:14:02.563370 21173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d8fd1ea29c44fa786a346f878aec471 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:14:02.563660 21180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d8fd1ea29c44fa786a346f878aec471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8fd1ea29c44fa786a346f878aec471" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38169 } }
I20260927 22:14:02.567504 21183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d8fd1ea29c44fa786a346f878aec471 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d8fd1ea29c44fa786a346f878aec471" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8fd1ea29c44fa786a346f878aec471" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38169 } } }
I20260927 22:14:02.567479 21184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d8fd1ea29c44fa786a346f878aec471 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d8fd1ea29c44fa786a346f878aec471. Latest consensus state: current_term: 1 leader_uuid: "7d8fd1ea29c44fa786a346f878aec471" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8fd1ea29c44fa786a346f878aec471" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38169 } } }
I20260927 22:14:02.568037 21183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d8fd1ea29c44fa786a346f878aec471 [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:02.568051 21184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d8fd1ea29c44fa786a346f878aec471 [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:02.581946 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:02.564356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41141 (local address 127.0.0.1:38169)
0927 22:14:02.564521 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:02.564529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:02.565189 (+ 660us) server_negotiation.cc:407] Connection header received
0927 22:14:02.565424 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:02.565429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:02.565521 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:02.565667 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:02.567845 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:02.569342 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:02.571836 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:02.572845 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:02.573003 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:02.576267 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:02.576272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:02.576275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:02.576308 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:02.579221 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:02.579923 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:02.579933 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:02.579935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:02.580046 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:02.580428 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:02.580433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:02.580436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:02.580760 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:02.581005 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:02.581346 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 22:14:02.581558 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:14:02.583776 21189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:14:02.589476 21189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:14:02.599305 21189 catalog_manager.cc:1347] Generated new cluster ID: c4d294e7ee1043fd81539a314ad0e824
I20260927 22:14:02.599427 21189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:14:02.613060 21189 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:14:02.614681 21189 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:14:02.628575 21189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d8fd1ea29c44fa786a346f878aec471: Generated new TSK 0
I20260927 22:14:02.629715 21189 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:14:02.638357 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 21106
I20260927 22:14:02.639707 21168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:14:02.731735 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 21106
2026-09-27T22:14:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2023 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:14:02 dist-test-slave-mp03 krb5kdc[21205](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:14:02 dist-test-slave-mp03 krb5kdc[21205](info): set up 2 sockets
Sep 27 22:14:02 dist-test-slave-mp03 krb5kdc[21205](info): commencing operation
krb5kdc: starting...
W20260927 22:14:03.846941 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:14:04 dist-test-slave-mp03 krb5kdc[21205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:14:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:14:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:14:04.101627 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45217
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:04.430354 21218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:04.430709 21218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:04.430820 21218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:04.438076 21218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:04.438174 21218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:04.438220 21218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:04.438292 21218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:04.438335 21218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:04.448000 21218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:04.449190 21218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:04.450448 21218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:04.462347 21223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:04.462368 21224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:04.462862 21226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:04.463225 21218 server_base.cc:1034] running on GCE node
I20260927 22:14:04.464010 21218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:04.465343 21218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:04.466550 21218 hybrid_clock.cc:648] HybridClock initialized: now 1790547244466499 us; error 42 us; skew 500 ppm
I20260927 22:14:04.466969 21218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:04 dist-test-slave-mp03 krb5kdc[21205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:04.470022 21218 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:14:04.471603 21218 webserver.cc:460] Webserver started at http://127.0.0.1:34723/ using document root <none> and password file <none>
I20260927 22:14:04.472289 21218 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:04.472440 21218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:04.472764 21218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:04.475768 21218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "14aab15610d94df3a36954503f5a751d"
format_stamp: "Formatted at 2026-09-27 22:14:04 on dist-test-slave-mp03"
server_key: "e287e047d5edfab1ee664118a9489ffb"
server_key_iv: "257d807043d12cedd8f0d60a06f536d7"
server_key_version: "encryptionkey@0"
I20260927 22:14:04.476527 21218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "14aab15610d94df3a36954503f5a751d"
format_stamp: "Formatted at 2026-09-27 22:14:04 on dist-test-slave-mp03"
server_key: "e287e047d5edfab1ee664118a9489ffb"
server_key_iv: "257d807043d12cedd8f0d60a06f536d7"
server_key_version: "encryptionkey@0"
I20260927 22:14:04.481590 21218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 22:14:04.485016 21233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:04.486414 21218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:14:04.486550 21218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "14aab15610d94df3a36954503f5a751d"
format_stamp: "Formatted at 2026-09-27 22:14:04 on dist-test-slave-mp03"
server_key: "e287e047d5edfab1ee664118a9489ffb"
server_key_iv: "257d807043d12cedd8f0d60a06f536d7"
server_key_version: "encryptionkey@0"
I20260927 22:14:04.486786 21218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:04.514050 21218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:04.518491 21218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:04.518787 21218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:04.537207 21218 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45217
I20260927 22:14:04.537261 21284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45217 every 8 connection(s)
I20260927 22:14:04.538573 21218 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:14:04.542435 21285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:04.545604 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 21218
I20260927 22:14:04.545826 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:14:04.546420 388 external_mini_cluster.cc:1428] Setting key c8adca6dffc7d09bc44c6b328362b5d1
I20260927 22:14:04.556512 21285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14aab15610d94df3a36954503f5a751d: Bootstrap starting.
I20260927 22:14:04.560485 21285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14aab15610d94df3a36954503f5a751d: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:04.561807 21285 log.cc:824] T 00000000000000000000000000000000 P 14aab15610d94df3a36954503f5a751d: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:04.562252 21288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:04.548561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42157 (local address 127.0.0.1:45217)
0927 22:14:04.549530 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:04.549546 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:04.549944 (+ 398us) server_negotiation.cc:407] Connection header received
0927 22:14:04.551833 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:04.551864 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:04.552475 (+ 611us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:04.552944 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:04.553465 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:04.555596 (+ 2131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:04.557025 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:04.558126 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:04.558355 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:04.559451 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:04.559462 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:04.559482 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:14:04.560199 (+ 717us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:04.560298 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:14:04.560341 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:04.560470 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:04.560632 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 22:14:04.560941 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":437,"threads_started":1}
I20260927 22:14:04.564903 21285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14aab15610d94df3a36954503f5a751d: No bootstrap required, opened a new log
I20260927 22:14:04.565966 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:14:04.573628 21285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14aab15610d94df3a36954503f5a751d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14aab15610d94df3a36954503f5a751d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45217 } }
I20260927 22:14:04.574056 21285 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14aab15610d94df3a36954503f5a751d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:04.574168 21285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14aab15610d94df3a36954503f5a751d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14aab15610d94df3a36954503f5a751d, State: Initialized, Role: FOLLOWER
I20260927 22:14:04.574813 21285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14aab15610d94df3a36954503f5a751d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14aab15610d94df3a36954503f5a751d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45217 } }
I20260927 22:14:04.575081 21285 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14aab15610d94df3a36954503f5a751d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:14:04.575273 21285 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14aab15610d94df3a36954503f5a751d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:14:04.575457 21285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14aab15610d94df3a36954503f5a751d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:04.577032 21288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:04.570661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57427 (local address 127.0.0.1:45217)
0927 22:14:04.570921 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:04.570929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:04.571323 (+ 394us) server_negotiation.cc:407] Connection header received
0927 22:14:04.571555 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:04.571559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:04.571619 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:04.571745 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:04.572200 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:04.572960 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:04.574379 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:04.574981 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:04.575117 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:04.576322 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:04.576333 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:04.576336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:14:04.576354 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:04.576394 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:14:04.576409 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:04.576515 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:04.576632 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 22:14:04.576737 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:14:04.577983 21285 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14aab15610d94df3a36954503f5a751d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14aab15610d94df3a36954503f5a751d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45217 } }
I20260927 22:14:04.578495 21285 leader_election.cc:302] T 00000000000000000000000000000000 P 14aab15610d94df3a36954503f5a751d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14aab15610d94df3a36954503f5a751d; no voters:
I20260927 22:14:04.579114 21285 leader_election.cc:288] T 00000000000000000000000000000000 P 14aab15610d94df3a36954503f5a751d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:14:04.579254 21295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14aab15610d94df3a36954503f5a751d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:04.579948 21295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14aab15610d94df3a36954503f5a751d [term 1 LEADER]: Becoming Leader. State: Replica: 14aab15610d94df3a36954503f5a751d, State: Running, Role: LEADER
I20260927 22:14:04.580534 21295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14aab15610d94df3a36954503f5a751d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14aab15610d94df3a36954503f5a751d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45217 } }
I20260927 22:14:04.581239 21285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14aab15610d94df3a36954503f5a751d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:14:04.583202 21297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14aab15610d94df3a36954503f5a751d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14aab15610d94df3a36954503f5a751d. Latest consensus state: current_term: 1 leader_uuid: "14aab15610d94df3a36954503f5a751d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14aab15610d94df3a36954503f5a751d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45217 } } }
I20260927 22:14:04.583600 21297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14aab15610d94df3a36954503f5a751d [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:04.584110 21296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14aab15610d94df3a36954503f5a751d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14aab15610d94df3a36954503f5a751d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14aab15610d94df3a36954503f5a751d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45217 } } }
I20260927 22:14:04.584446 21296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14aab15610d94df3a36954503f5a751d [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:04.584476 21300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:14:04.589561 21300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:14:04.601110 21300 catalog_manager.cc:1347] Generated new cluster ID: ce9eca533d5a45ecb834e7f2ac746448
I20260927 22:14:04.601253 21300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:14:04.638885 21300 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:14:04.640230 21300 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:14:04.647329 21300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14aab15610d94df3a36954503f5a751d: Generated new TSK 0
I20260927 22:14:04.648202 21300 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:14:04.658197 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 21218
W20260927 22:14:04.659560 21280 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42157
W20260927 22:14:04.660277 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 21218: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:14:04.660405 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 21218
2026-09-27T22:14:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1928 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:14:04 dist-test-slave-mp03 krb5kdc[21316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:14:04 dist-test-slave-mp03 krb5kdc[21316](info): set up 2 sockets
Sep 27 22:14:04 dist-test-slave-mp03 krb5kdc[21316](info): commencing operation
krb5kdc: starting...
W20260927 22:14:05.766909 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:14:05 dist-test-slave-mp03 krb5kdc[21316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:14:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:14:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:14:06.021734 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38633
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:06.350806 21329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:06.351178 21329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:06.351337 21329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:06.358861 21329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:06.359161 21329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:06.359308 21329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:06.359418 21329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:06.359529 21329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:06.370278 21329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:06.371634 21329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:06.372920 21329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:06.384840 21334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:06.384812 21335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:06.384919 21337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:06.385761 21329 server_base.cc:1034] running on GCE node
I20260927 22:14:06.386631 21329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:06.388110 21329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:06.389348 21329 hybrid_clock.cc:648] HybridClock initialized: now 1790547246389289 us; error 54 us; skew 500 ppm
I20260927 22:14:06.389698 21329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:06 dist-test-slave-mp03 krb5kdc[21316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:06.398288 21329 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:14:06.399973 21329 webserver.cc:460] Webserver started at http://127.0.0.1:46319/ using document root <none> and password file <none>
I20260927 22:14:06.400650 21329 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:06.400750 21329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:06.401019 21329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:06.404119 21329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "35dc9d2c503c48c689d05a89b8ad157f"
format_stamp: "Formatted at 2026-09-27 22:14:06 on dist-test-slave-mp03"
server_key: "2118c43141db6012fa290c17b63bb2f1"
server_key_iv: "31ce0548745f0dbc76af1c64de60e654"
server_key_version: "encryptionkey@0"
I20260927 22:14:06.404912 21329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "35dc9d2c503c48c689d05a89b8ad157f"
format_stamp: "Formatted at 2026-09-27 22:14:06 on dist-test-slave-mp03"
server_key: "2118c43141db6012fa290c17b63bb2f1"
server_key_iv: "31ce0548745f0dbc76af1c64de60e654"
server_key_version: "encryptionkey@0"
I20260927 22:14:06.410269 21329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:14:06.413995 21344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:06.415757 21329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:14:06.415977 21329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "35dc9d2c503c48c689d05a89b8ad157f"
format_stamp: "Formatted at 2026-09-27 22:14:06 on dist-test-slave-mp03"
server_key: "2118c43141db6012fa290c17b63bb2f1"
server_key_iv: "31ce0548745f0dbc76af1c64de60e654"
server_key_version: "encryptionkey@0"
I20260927 22:14:06.416203 21329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:06.422421 21329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:06.426707 21329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:06.427028 21329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:06.446044 21329 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38633
I20260927 22:14:06.446102 21395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38633 every 8 connection(s)
I20260927 22:14:06.447497 21329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:14:06.451231 21396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:06.455869 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 21329
I20260927 22:14:06.456053 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:14:06.456590 388 external_mini_cluster.cc:1428] Setting key 0b32ee1b6bf14a38d003263d9c1198db
I20260927 22:14:06.462776 21396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35dc9d2c503c48c689d05a89b8ad157f: Bootstrap starting.
I20260927 22:14:06.464603 21399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:06.458385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33591 (local address 127.0.0.1:38633)
0927 22:14:06.459269 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:06.459282 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:06.459323 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:14:06.461215 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:06.461238 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:06.461579 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:06.461915 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:06.463010 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:06.463017 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:06.463029 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:14:06.463045 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:06.463109 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:14:06.463135 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:06.463307 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:06.463453 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 22:14:06.463662 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":451,"threads_started":1}
I20260927 22:14:06.466034 21396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35dc9d2c503c48c689d05a89b8ad157f: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:06.467228 21396 log.cc:824] T 00000000000000000000000000000000 P 35dc9d2c503c48c689d05a89b8ad157f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:06.468467 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:14:06.470551 21396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35dc9d2c503c48c689d05a89b8ad157f: No bootstrap required, opened a new log
I20260927 22:14:06.476397 21399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:06.473578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42649 (local address 127.0.0.1:38633)
0927 22:14:06.473773 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:06.473780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:06.474243 (+ 463us) server_negotiation.cc:407] Connection header received
0927 22:14:06.474443 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:06.474450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:06.474513 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:06.474654 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:06.475720 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:06.475731 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:06.475736 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:14:06.475754 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:06.475801 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:14:06.475827 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:06.475947 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:06.476036 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 22:14:06.476092 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:14:06.479936 21396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35dc9d2c503c48c689d05a89b8ad157f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dc9d2c503c48c689d05a89b8ad157f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38633 } }
I20260927 22:14:06.480382 21396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35dc9d2c503c48c689d05a89b8ad157f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:06.480485 21396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35dc9d2c503c48c689d05a89b8ad157f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35dc9d2c503c48c689d05a89b8ad157f, State: Initialized, Role: FOLLOWER
I20260927 22:14:06.481086 21396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35dc9d2c503c48c689d05a89b8ad157f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dc9d2c503c48c689d05a89b8ad157f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38633 } }
I20260927 22:14:06.481309 21396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35dc9d2c503c48c689d05a89b8ad157f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:14:06.481477 21396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35dc9d2c503c48c689d05a89b8ad157f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:14:06.481650 21396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35dc9d2c503c48c689d05a89b8ad157f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:06.483952 21396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35dc9d2c503c48c689d05a89b8ad157f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dc9d2c503c48c689d05a89b8ad157f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38633 } }
I20260927 22:14:06.484488 21396 leader_election.cc:302] T 00000000000000000000000000000000 P 35dc9d2c503c48c689d05a89b8ad157f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35dc9d2c503c48c689d05a89b8ad157f; no voters:
I20260927 22:14:06.485090 21396 leader_election.cc:288] T 00000000000000000000000000000000 P 35dc9d2c503c48c689d05a89b8ad157f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:14:06.485270 21406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35dc9d2c503c48c689d05a89b8ad157f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:06.487037 21406 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35dc9d2c503c48c689d05a89b8ad157f [term 1 LEADER]: Becoming Leader. State: Replica: 35dc9d2c503c48c689d05a89b8ad157f, State: Running, Role: LEADER
I20260927 22:14:06.487360 21396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35dc9d2c503c48c689d05a89b8ad157f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:14:06.487617 21406 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35dc9d2c503c48c689d05a89b8ad157f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dc9d2c503c48c689d05a89b8ad157f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38633 } }
I20260927 22:14:06.490520 21408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35dc9d2c503c48c689d05a89b8ad157f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35dc9d2c503c48c689d05a89b8ad157f. Latest consensus state: current_term: 1 leader_uuid: "35dc9d2c503c48c689d05a89b8ad157f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dc9d2c503c48c689d05a89b8ad157f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38633 } } }
I20260927 22:14:06.490638 21407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35dc9d2c503c48c689d05a89b8ad157f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35dc9d2c503c48c689d05a89b8ad157f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35dc9d2c503c48c689d05a89b8ad157f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38633 } } }
I20260927 22:14:06.490983 21408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35dc9d2c503c48c689d05a89b8ad157f [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:06.491025 21407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35dc9d2c503c48c689d05a89b8ad157f [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:06.494647 21414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:14:06.500766 21414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:14:06.509058 21414 catalog_manager.cc:1347] Generated new cluster ID: b821ab3608404dd685798599a5781de2
I20260927 22:14:06.509171 21414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:14:06.545814 21414 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:14:06.546348 21414 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:14:06.557680 21414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35dc9d2c503c48c689d05a89b8ad157f: Generated new TSK 0
I20260927 22:14:06.558802 21414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:14:06.570560 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 21329
W20260927 22:14:06.572022 21391 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33591
W20260927 22:14:06.572796 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 21329: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:14:06.572956 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 21329
2026-09-27T22:14:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1910 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:14:06 dist-test-slave-mp03 krb5kdc[21427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:14:06 dist-test-slave-mp03 krb5kdc[21427](info): set up 2 sockets
Sep 27 22:14:06 dist-test-slave-mp03 krb5kdc[21427](info): commencing operation
krb5kdc: starting...
W20260927 22:14:07.682965 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:14:07 dist-test-slave-mp03 krb5kdc[21427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:14:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:14:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:14:07.949424 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33991
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:08.277182 21440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:08.277469 21440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:08.277561 21440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:08.284773 21440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:08.284891 21440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:08.284943 21440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:08.285006 21440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:08.285050 21440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:08.294986 21440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:08.296228 21440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:08.297578 21440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:08.309726 21445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:08.309880 21448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:08.309696 21446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:08.310468 21440 server_base.cc:1034] running on GCE node
I20260927 22:14:08.311262 21440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:08.312719 21440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:08.313931 21440 hybrid_clock.cc:648] HybridClock initialized: now 1790547248313896 us; error 32 us; skew 500 ppm
I20260927 22:14:08.314307 21440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:08 dist-test-slave-mp03 krb5kdc[21427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:08.317541 21440 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:14:08.319124 21440 webserver.cc:460] Webserver started at http://127.0.0.1:45163/ using document root <none> and password file <none>
I20260927 22:14:08.319845 21440 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:08.319993 21440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:08.320313 21440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:08.323393 21440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "8c33419eb033411992b8d9fe1e7cce6d"
format_stamp: "Formatted at 2026-09-27 22:14:08 on dist-test-slave-mp03"
server_key: "c372c3ea16b88da48e2b560c063e9f71"
server_key_iv: "f6ab25fbdfc84416bbfaa492ec2962e0"
server_key_version: "encryptionkey@0"
I20260927 22:14:08.324186 21440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "8c33419eb033411992b8d9fe1e7cce6d"
format_stamp: "Formatted at 2026-09-27 22:14:08 on dist-test-slave-mp03"
server_key: "c372c3ea16b88da48e2b560c063e9f71"
server_key_iv: "f6ab25fbdfc84416bbfaa492ec2962e0"
server_key_version: "encryptionkey@0"
I20260927 22:14:08.329568 21440 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:14:08.333122 21455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:08.334518 21440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:14:08.334733 21440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "8c33419eb033411992b8d9fe1e7cce6d"
format_stamp: "Formatted at 2026-09-27 22:14:08 on dist-test-slave-mp03"
server_key: "c372c3ea16b88da48e2b560c063e9f71"
server_key_iv: "f6ab25fbdfc84416bbfaa492ec2962e0"
server_key_version: "encryptionkey@0"
I20260927 22:14:08.334946 21440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:08.352496 21440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:08.356573 21440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:08.356870 21440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:08.375141 21440 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33991
I20260927 22:14:08.375165 21506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33991 every 8 connection(s)
I20260927 22:14:08.376572 21440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:14:08.380303 21507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:08.383625 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 21440
I20260927 22:14:08.383806 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:14:08.384222 388 external_mini_cluster.cc:1428] Setting key e958e9c03c92a78ea4017c262c14b55b
I20260927 22:14:08.393237 21507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c33419eb033411992b8d9fe1e7cce6d: Bootstrap starting.
I20260927 22:14:08.396796 21507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c33419eb033411992b8d9fe1e7cce6d: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:08.397773 21507 log.cc:824] T 00000000000000000000000000000000 P 8c33419eb033411992b8d9fe1e7cce6d: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:14:08 dist-test-slave-mp03 krb5kdc[21427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:14:08.400913 21507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c33419eb033411992b8d9fe1e7cce6d: No bootstrap required, opened a new log
I20260927 22:14:08.410229 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:08.386075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:44079 (local address 127.0.0.1:33991)
0927 22:14:08.386815 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:08.386830 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:08.386882 (+ 52us) server_negotiation.cc:407] Connection header received
0927 22:14:08.388985 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:08.389016 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:08.389583 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:08.390021 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:08.391380 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:08.393339 (+ 1959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:08.394768 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:08.395798 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:08.395984 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:08.400457 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:08.400470 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:08.400489 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:08.400532 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:08.404644 (+ 4112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:08.405787 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:08.405802 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:08.405819 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:08.406156 (+ 337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:08.406904 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:08.406910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:08.406914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:08.407591 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:08.408001 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:08.408553 (+ 552us) server_negotiation.cc:299] Negotiation successful
0927 22:14:08.408938 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":296,"threads_started":1}
I20260927 22:14:08.411010 21507 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c33419eb033411992b8d9fe1e7cce6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c33419eb033411992b8d9fe1e7cce6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33991 } }
I20260927 22:14:08.411427 21507 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c33419eb033411992b8d9fe1e7cce6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:08.411546 21507 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c33419eb033411992b8d9fe1e7cce6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c33419eb033411992b8d9fe1e7cce6d, State: Initialized, Role: FOLLOWER
I20260927 22:14:08.412176 21507 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c33419eb033411992b8d9fe1e7cce6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c33419eb033411992b8d9fe1e7cce6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33991 } }
I20260927 22:14:08.412458 21507 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c33419eb033411992b8d9fe1e7cce6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:14:08.412616 21507 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c33419eb033411992b8d9fe1e7cce6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:14:08.412808 21507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c33419eb033411992b8d9fe1e7cce6d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:08.415216 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:14:08.415733 21507 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c33419eb033411992b8d9fe1e7cce6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c33419eb033411992b8d9fe1e7cce6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33991 } }
I20260927 22:14:08.416275 21507 leader_election.cc:302] T 00000000000000000000000000000000 P 8c33419eb033411992b8d9fe1e7cce6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c33419eb033411992b8d9fe1e7cce6d; no voters:
I20260927 22:14:08.416926 21507 leader_election.cc:288] T 00000000000000000000000000000000 P 8c33419eb033411992b8d9fe1e7cce6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:14:08.417093 21512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c33419eb033411992b8d9fe1e7cce6d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:08.419063 21507 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c33419eb033411992b8d9fe1e7cce6d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:14:08.419306 21512 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c33419eb033411992b8d9fe1e7cce6d [term 1 LEADER]: Becoming Leader. State: Replica: 8c33419eb033411992b8d9fe1e7cce6d, State: Running, Role: LEADER
I20260927 22:14:08.420193 21512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c33419eb033411992b8d9fe1e7cce6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c33419eb033411992b8d9fe1e7cce6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33991 } }
I20260927 22:14:08.424932 21514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c33419eb033411992b8d9fe1e7cce6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c33419eb033411992b8d9fe1e7cce6d. Latest consensus state: current_term: 1 leader_uuid: "8c33419eb033411992b8d9fe1e7cce6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c33419eb033411992b8d9fe1e7cce6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33991 } } }
I20260927 22:14:08.425352 21514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c33419eb033411992b8d9fe1e7cce6d [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:08.426033 21513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c33419eb033411992b8d9fe1e7cce6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c33419eb033411992b8d9fe1e7cce6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c33419eb033411992b8d9fe1e7cce6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33991 } } }
I20260927 22:14:08.426364 21513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c33419eb033411992b8d9fe1e7cce6d [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:08.437453 21527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:14:08.442855 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:08.423737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:57741 (local address 127.0.0.1:33991)
0927 22:14:08.424600 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:08.424606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:08.424626 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:14:08.424688 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:08.424695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:08.424799 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:08.424925 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:08.426801 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:08.428360 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:08.431028 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:08.432016 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:08.432183 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:08.436007 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:08.436016 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:08.436023 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:08.436077 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:08.439940 (+ 3863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:08.440707 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:08.440713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:08.440718 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:08.440846 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:08.441312 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:08.441317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:08.441322 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:08.441611 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:08.441884 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:08.442262 (+ 378us) server_negotiation.cc:299] Negotiation successful
0927 22:14:08.442447 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":92}
I20260927 22:14:08.449707 21527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:14:08.461305 21527 catalog_manager.cc:1347] Generated new cluster ID: cc6a70ecaff04eb3b68d88089ce5e38c
I20260927 22:14:08.461447 21527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:14:08.478179 21527 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:14:08.480082 21527 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:14:08.496773 21527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c33419eb033411992b8d9fe1e7cce6d: Generated new TSK 0
I20260927 22:14:08.498016 21527 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:14:08.513435 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 21440
I20260927 22:14:08.514742 21502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:14:08.609913 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 21440
W20260927 22:14:08.624651 21425 connection.cc:535] client connection to 127.0.0.1:33991 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33991): Connection reset by peer (error 104)
2026-09-27T22:14:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2036 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:14:08 dist-test-slave-mp03 krb5kdc[21539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:14:08 dist-test-slave-mp03 krb5kdc[21539](info): set up 2 sockets
Sep 27 22:14:08 dist-test-slave-mp03 krb5kdc[21539](info): commencing operation
krb5kdc: starting...
W20260927 22:14:09.715009 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:14:09 dist-test-slave-mp03 krb5kdc[21539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:14:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:14:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:14:09.977687 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46061
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:10.305928 21552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:10.306214 21552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:10.306382 21552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:10.313541 21552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:10.313669 21552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:10.313769 21552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:10.313884 21552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:10.313964 21552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:10.323796 21552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:10.325093 21552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:10.326413 21552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:10.338578 21557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:10.338541 21560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:10.338626 21558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:10.339537 21552 server_base.cc:1034] running on GCE node
I20260927 22:14:10.340307 21552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:10.341835 21552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:10.343128 21552 hybrid_clock.cc:648] HybridClock initialized: now 1790547250343093 us; error 29 us; skew 500 ppm
I20260927 22:14:10.343500 21552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:10 dist-test-slave-mp03 krb5kdc[21539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:10.346951 21552 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:14:10.348520 21552 webserver.cc:460] Webserver started at http://127.0.0.1:34635/ using document root <none> and password file <none>
I20260927 22:14:10.349191 21552 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:10.349318 21552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:10.349641 21552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:10.352840 21552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "0094d121b302430aa4c574d44298ac72"
format_stamp: "Formatted at 2026-09-27 22:14:10 on dist-test-slave-mp03"
server_key: "ec6b72d97160817df200732da07ec141"
server_key_iv: "f09f984ce97818d278d7e5422be0cfaa"
server_key_version: "encryptionkey@0"
I20260927 22:14:10.353626 21552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "0094d121b302430aa4c574d44298ac72"
format_stamp: "Formatted at 2026-09-27 22:14:10 on dist-test-slave-mp03"
server_key: "ec6b72d97160817df200732da07ec141"
server_key_iv: "f09f984ce97818d278d7e5422be0cfaa"
server_key_version: "encryptionkey@0"
I20260927 22:14:10.358871 21552 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:14:10.362403 21567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:10.363840 21552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:14:10.364040 21552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "0094d121b302430aa4c574d44298ac72"
format_stamp: "Formatted at 2026-09-27 22:14:10 on dist-test-slave-mp03"
server_key: "ec6b72d97160817df200732da07ec141"
server_key_iv: "f09f984ce97818d278d7e5422be0cfaa"
server_key_version: "encryptionkey@0"
I20260927 22:14:10.364246 21552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:10.380086 21552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:10.384536 21552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:10.384838 21552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:10.403856 21552 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46061
I20260927 22:14:10.403975 21618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46061 every 8 connection(s)
I20260927 22:14:10.405269 21552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:14:10.409091 21619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:10.412846 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 21552
I20260927 22:14:10.413048 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:14:10.413614 388 external_mini_cluster.cc:1428] Setting key c64158f35b4aab57d82a59078a54eb6b
I20260927 22:14:10.424134 21619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0094d121b302430aa4c574d44298ac72: Bootstrap starting.
I20260927 22:14:10.428174 21619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0094d121b302430aa4c574d44298ac72: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:10.429493 21619 log.cc:824] T 00000000000000000000000000000000 P 0094d121b302430aa4c574d44298ac72: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:10.429746 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:10.415962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:45837 (local address 127.0.0.1:46061)
0927 22:14:10.416890 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:10.416908 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:10.417378 (+ 470us) server_negotiation.cc:407] Connection header received
0927 22:14:10.419525 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:10.419555 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:10.420135 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:10.420608 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:10.421141 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:10.423181 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:10.424582 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:10.425569 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:10.425772 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:10.426884 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:10.426896 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:10.426913 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:14:10.427664 (+ 751us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:10.427769 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:14:10.427800 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:10.427972 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:10.428187 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 22:14:10.428389 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":389,"threads_started":1}
I20260927 22:14:10.432742 21619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0094d121b302430aa4c574d44298ac72: No bootstrap required, opened a new log
I20260927 22:14:10.433753 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:14:10.442953 21619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0094d121b302430aa4c574d44298ac72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0094d121b302430aa4c574d44298ac72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46061 } }
I20260927 22:14:10.443436 21619 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0094d121b302430aa4c574d44298ac72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:10.443557 21619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0094d121b302430aa4c574d44298ac72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0094d121b302430aa4c574d44298ac72, State: Initialized, Role: FOLLOWER
I20260927 22:14:10.444326 21619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0094d121b302430aa4c574d44298ac72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0094d121b302430aa4c574d44298ac72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46061 } }
I20260927 22:14:10.444603 21619 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0094d121b302430aa4c574d44298ac72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:14:10.444790 21619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0094d121b302430aa4c574d44298ac72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:14:10.444988 21619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0094d121b302430aa4c574d44298ac72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:10.445803 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:10.438961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:34197 (local address 127.0.0.1:46061)
0927 22:14:10.439153 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:10.439159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:10.439613 (+ 454us) server_negotiation.cc:407] Connection header received
0927 22:14:10.439761 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:10.439766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:10.439853 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:10.439967 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:10.440455 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:10.441671 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:10.443064 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:10.443719 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:10.443892 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:10.445009 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:10.445014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:10.445017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:14:10.445035 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:10.445125 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:14:10.445146 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:10.445279 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:10.445452 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 22:14:10.445534 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:14:10.447890 21619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0094d121b302430aa4c574d44298ac72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0094d121b302430aa4c574d44298ac72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46061 } }
I20260927 22:14:10.448397 21619 leader_election.cc:302] T 00000000000000000000000000000000 P 0094d121b302430aa4c574d44298ac72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0094d121b302430aa4c574d44298ac72; no voters:
I20260927 22:14:10.448930 21619 leader_election.cc:288] T 00000000000000000000000000000000 P 0094d121b302430aa4c574d44298ac72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:14:10.449064 21629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0094d121b302430aa4c574d44298ac72 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:10.449806 21629 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0094d121b302430aa4c574d44298ac72 [term 1 LEADER]: Becoming Leader. State: Replica: 0094d121b302430aa4c574d44298ac72, State: Running, Role: LEADER
I20260927 22:14:10.450403 21629 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0094d121b302430aa4c574d44298ac72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0094d121b302430aa4c574d44298ac72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46061 } }
I20260927 22:14:10.451212 21619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0094d121b302430aa4c574d44298ac72 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:14:10.453267 21631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0094d121b302430aa4c574d44298ac72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0094d121b302430aa4c574d44298ac72. Latest consensus state: current_term: 1 leader_uuid: "0094d121b302430aa4c574d44298ac72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0094d121b302430aa4c574d44298ac72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46061 } } }
I20260927 22:14:10.453240 21630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0094d121b302430aa4c574d44298ac72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0094d121b302430aa4c574d44298ac72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0094d121b302430aa4c574d44298ac72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46061 } } }
I20260927 22:14:10.453744 21630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0094d121b302430aa4c574d44298ac72 [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:10.453732 21631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0094d121b302430aa4c574d44298ac72 [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:10.454757 21634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:14:10.460772 21634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:14:10.472108 21634 catalog_manager.cc:1347] Generated new cluster ID: 0a4b0a2e6f864869b48b45aeb6a0d49c
I20260927 22:14:10.472236 21634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:14:10.502828 21634 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:14:10.504814 21634 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:14:10.518253 21634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0094d121b302430aa4c574d44298ac72: Generated new TSK 0
I20260927 22:14:10.519373 21634 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:14:10.531502 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 21552
W20260927 22:14:10.533334 21614 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.82:45837
W20260927 22:14:10.534127 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 21552: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:14:10.534276 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 21552
W20260927 22:14:10.551482 21537 connection.cc:535] client connection to 127.0.0.1:46061 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46061): Connection reset by peer (error 104)
2026-09-27T22:14:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1927 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:14:10 dist-test-slave-mp03 krb5kdc[21650](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:14:10 dist-test-slave-mp03 krb5kdc[21650](info): set up 2 sockets
Sep 27 22:14:10 dist-test-slave-mp03 krb5kdc[21650](info): commencing operation
krb5kdc: starting...
W20260927 22:14:11.643041 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.011s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:14:11 dist-test-slave-mp03 krb5kdc[21650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:14:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:14:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:14:11.910732 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38685
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:12.253724 21663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:12.254217 21663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:12.254403 21663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:12.262019 21663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:12.262202 21663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:12.262320 21663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:12.262441 21663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:12.262534 21663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:12.273109 21663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:12.274693 21663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:12.276119 21663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:12.288790 21671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:12.288770 21668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:12.288736 21669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:12.289734 21663 server_base.cc:1034] running on GCE node
I20260927 22:14:12.290519 21663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:12.292027 21663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:12.293267 21663 hybrid_clock.cc:648] HybridClock initialized: now 1790547252293222 us; error 49 us; skew 500 ppm
I20260927 22:14:12.293627 21663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:12 dist-test-slave-mp03 krb5kdc[21650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:12.304744 21663 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:14:12.306392 21663 webserver.cc:460] Webserver started at http://127.0.0.1:37371/ using document root <none> and password file <none>
I20260927 22:14:12.307256 21663 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:12.307428 21663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:12.307751 21663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:12.310900 21663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "672d47a345504d6f88d5ebbe29ee1dde"
format_stamp: "Formatted at 2026-09-27 22:14:12 on dist-test-slave-mp03"
server_key: "bd36d05f604806c558837bb4474ff4f3"
server_key_iv: "897ce80bde59250adb4ba43c0795ffd2"
server_key_version: "encryptionkey@0"
I20260927 22:14:12.311690 21663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "672d47a345504d6f88d5ebbe29ee1dde"
format_stamp: "Formatted at 2026-09-27 22:14:12 on dist-test-slave-mp03"
server_key: "bd36d05f604806c558837bb4474ff4f3"
server_key_iv: "897ce80bde59250adb4ba43c0795ffd2"
server_key_version: "encryptionkey@0"
I20260927 22:14:12.317108 21663 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:14:12.321103 21678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:12.322570 21663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:14:12.322860 21663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "672d47a345504d6f88d5ebbe29ee1dde"
format_stamp: "Formatted at 2026-09-27 22:14:12 on dist-test-slave-mp03"
server_key: "bd36d05f604806c558837bb4474ff4f3"
server_key_iv: "897ce80bde59250adb4ba43c0795ffd2"
server_key_version: "encryptionkey@0"
I20260927 22:14:12.323092 21663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:12.329360 21663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:12.333788 21663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:12.334187 21663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:12.354359 21663 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38685
I20260927 22:14:12.354401 21729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38685 every 8 connection(s)
I20260927 22:14:12.356034 21663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:14:12.359774 21730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:12.366091 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 21663
I20260927 22:14:12.366281 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:14:12.366806 388 external_mini_cluster.cc:1428] Setting key 971cfa754a622cef72a9519e6d65ded9
I20260927 22:14:12.370524 21730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 672d47a345504d6f88d5ebbe29ee1dde: Bootstrap starting.
I20260927 22:14:12.374572 21730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 672d47a345504d6f88d5ebbe29ee1dde: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:12.375960 21733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:12.368726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:57721 (local address 127.0.0.1:38685)
0927 22:14:12.369580 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:12.369595 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:12.369643 (+ 48us) server_negotiation.cc:407] Connection header received
0927 22:14:12.372079 (+ 2436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:12.372109 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:12.372596 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:12.373042 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:12.373962 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:12.373969 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:12.373985 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:14:12.374000 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:12.374081 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:14:12.374106 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:12.374287 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:12.374409 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 22:14:12.374632 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":369,"threads_started":1}
I20260927 22:14:12.376047 21730 log.cc:824] T 00000000000000000000000000000000 P 672d47a345504d6f88d5ebbe29ee1dde: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:12.379470 21730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 672d47a345504d6f88d5ebbe29ee1dde: No bootstrap required, opened a new log
I20260927 22:14:12.380378 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:14:12.387393 21733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:12.384920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:45095 (local address 127.0.0.1:38685)
0927 22:14:12.385115 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:12.385121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:12.385455 (+ 334us) server_negotiation.cc:407] Connection header received
0927 22:14:12.385629 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:12.385632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:12.385683 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:12.385767 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:12.386722 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:12.386727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:12.386730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:14:12.386744 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:12.386782 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:14:12.386796 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:12.386890 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:12.387026 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 22:14:12.387077 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:14:12.388155 21730 raft_consensus.cc:348] T 00000000000000000000000000000000 P 672d47a345504d6f88d5ebbe29ee1dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672d47a345504d6f88d5ebbe29ee1dde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38685 } }
I20260927 22:14:12.388657 21730 raft_consensus.cc:374] T 00000000000000000000000000000000 P 672d47a345504d6f88d5ebbe29ee1dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:12.388772 21730 raft_consensus.cc:729] T 00000000000000000000000000000000 P 672d47a345504d6f88d5ebbe29ee1dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672d47a345504d6f88d5ebbe29ee1dde, State: Initialized, Role: FOLLOWER
I20260927 22:14:12.389370 21730 consensus_queue.cc:260] T 00000000000000000000000000000000 P 672d47a345504d6f88d5ebbe29ee1dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672d47a345504d6f88d5ebbe29ee1dde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38685 } }
I20260927 22:14:12.389598 21730 raft_consensus.cc:388] T 00000000000000000000000000000000 P 672d47a345504d6f88d5ebbe29ee1dde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:14:12.389698 21730 raft_consensus.cc:482] T 00000000000000000000000000000000 P 672d47a345504d6f88d5ebbe29ee1dde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:14:12.389827 21730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 672d47a345504d6f88d5ebbe29ee1dde [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:12.392076 21730 raft_consensus.cc:504] T 00000000000000000000000000000000 P 672d47a345504d6f88d5ebbe29ee1dde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672d47a345504d6f88d5ebbe29ee1dde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38685 } }
I20260927 22:14:12.392567 21730 leader_election.cc:302] T 00000000000000000000000000000000 P 672d47a345504d6f88d5ebbe29ee1dde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 672d47a345504d6f88d5ebbe29ee1dde; no voters:
I20260927 22:14:12.393060 21730 leader_election.cc:288] T 00000000000000000000000000000000 P 672d47a345504d6f88d5ebbe29ee1dde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:14:12.393215 21740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 672d47a345504d6f88d5ebbe29ee1dde [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:12.394435 21740 raft_consensus.cc:686] T 00000000000000000000000000000000 P 672d47a345504d6f88d5ebbe29ee1dde [term 1 LEADER]: Becoming Leader. State: Replica: 672d47a345504d6f88d5ebbe29ee1dde, State: Running, Role: LEADER
I20260927 22:14:12.394968 21740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 672d47a345504d6f88d5ebbe29ee1dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672d47a345504d6f88d5ebbe29ee1dde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38685 } }
I20260927 22:14:12.395325 21730 sys_catalog.cc:564] T 00000000000000000000000000000000 P 672d47a345504d6f88d5ebbe29ee1dde [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:14:12.397648 21742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 672d47a345504d6f88d5ebbe29ee1dde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 672d47a345504d6f88d5ebbe29ee1dde. Latest consensus state: current_term: 1 leader_uuid: "672d47a345504d6f88d5ebbe29ee1dde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672d47a345504d6f88d5ebbe29ee1dde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38685 } } }
I20260927 22:14:12.397805 21741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 672d47a345504d6f88d5ebbe29ee1dde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "672d47a345504d6f88d5ebbe29ee1dde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672d47a345504d6f88d5ebbe29ee1dde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38685 } } }
I20260927 22:14:12.398187 21741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 672d47a345504d6f88d5ebbe29ee1dde [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:12.398072 21742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 672d47a345504d6f88d5ebbe29ee1dde [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:12.400182 21748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:14:12.405855 21748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:14:12.415355 21748 catalog_manager.cc:1347] Generated new cluster ID: d3cb1cad7ce444db9068336965f5c06c
I20260927 22:14:12.415506 21748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:14:12.450865 21748 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:14:12.451350 21748 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:14:12.457504 21748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 672d47a345504d6f88d5ebbe29ee1dde: Generated new TSK 0
I20260927 22:14:12.458323 21748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:14:12.470150 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 21663
W20260927 22:14:12.471344 21725 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.82:57721
W20260927 22:14:12.472114 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 21663: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:14:12.472258 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 21663
2026-09-27T22:14:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1937 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:14:12 dist-test-slave-mp03 krb5kdc[21761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:14:12 dist-test-slave-mp03 krb5kdc[21761](info): set up 2 sockets
Sep 27 22:14:12 dist-test-slave-mp03 krb5kdc[21761](info): commencing operation
krb5kdc: starting...
W20260927 22:14:13.583205 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:14:13 dist-test-slave-mp03 krb5kdc[21761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:14:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:14:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:14:13.843397 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35405
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:14.171332 21774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:14.171634 21774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:14.171829 21774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:14.179088 21774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:14.179193 21774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:14.179301 21774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:14.179412 21774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:14.179502 21774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:14.190070 21774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35405
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:14.191506 21774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:14.192802 21774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:14.204658 21779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:14.204705 21780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:14.204741 21782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:14.205561 21774 server_base.cc:1034] running on GCE node
I20260927 22:14:14.206334 21774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:14.207893 21774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:14.209148 21774 hybrid_clock.cc:648] HybridClock initialized: now 1790547254209112 us; error 51 us; skew 500 ppm
I20260927 22:14:14.209534 21774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:14 dist-test-slave-mp03 krb5kdc[21761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:14.213027 21774 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:14:14.214687 21774 webserver.cc:460] Webserver started at http://127.0.0.1:41689/ using document root <none> and password file <none>
I20260927 22:14:14.215391 21774 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:14.215539 21774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:14.215858 21774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:14.219012 21774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "311a8c41efe847389bc0c6816e125fff"
format_stamp: "Formatted at 2026-09-27 22:14:14 on dist-test-slave-mp03"
server_key: "3999aff1d57418691aeb63376fb75dbc"
server_key_iv: "c9b747f950ec53fa5963ede22ce26061"
server_key_version: "encryptionkey@0"
I20260927 22:14:14.219794 21774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "311a8c41efe847389bc0c6816e125fff"
format_stamp: "Formatted at 2026-09-27 22:14:14 on dist-test-slave-mp03"
server_key: "3999aff1d57418691aeb63376fb75dbc"
server_key_iv: "c9b747f950ec53fa5963ede22ce26061"
server_key_version: "encryptionkey@0"
I20260927 22:14:14.225071 21774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 22:14:14.228667 21789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:14.230083 21774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:14:14.230273 21774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "311a8c41efe847389bc0c6816e125fff"
format_stamp: "Formatted at 2026-09-27 22:14:14 on dist-test-slave-mp03"
server_key: "3999aff1d57418691aeb63376fb75dbc"
server_key_iv: "c9b747f950ec53fa5963ede22ce26061"
server_key_version: "encryptionkey@0"
I20260927 22:14:14.230472 21774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:14.260169 21774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:14.264750 21774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:14.265044 21774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:14.284626 21774 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35405
I20260927 22:14:14.284670 21840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35405 every 8 connection(s)
I20260927 22:14:14.286015 21774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:14:14.288900 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 21774
I20260927 22:14:14.289103 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:14:14.289618 388 external_mini_cluster.cc:1428] Setting key 13b385dbff5e324330c1491d459d7796
I20260927 22:14:14.290421 21841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:14:14 dist-test-slave-mp03 krb5kdc[21761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:14:14.303004 21841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 311a8c41efe847389bc0c6816e125fff: Bootstrap starting.
I20260927 22:14:14.306349 21841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 311a8c41efe847389bc0c6816e125fff: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:14.307487 21841 log.cc:824] T 00000000000000000000000000000000 P 311a8c41efe847389bc0c6816e125fff: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:14.310986 21841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 311a8c41efe847389bc0c6816e125fff: No bootstrap required, opened a new log
I20260927 22:14:14.311962 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:14.291454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:49525 (local address 127.0.0.1:35405)
0927 22:14:14.292206 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:14.292218 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:14.292258 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:14:14.293838 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:14.293859 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:14.294278 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:14.294734 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:14.296181 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:14.297514 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:14.298897 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:14.299596 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:14.299819 (+ 223us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:14.304205 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:14.304216 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:14.304230 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:14.304275 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:14.307490 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:14.308516 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:14.308525 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:14.308534 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:14.308756 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:14.309267 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:14.309271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:14.309273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:14.309774 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:14.310012 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:14.310545 (+ 533us) server_negotiation.cc:299] Negotiation successful
0927 22:14:14.310920 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":318,"threads_started":1}
I20260927 22:14:14.315768 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:14:14.320482 21841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 311a8c41efe847389bc0c6816e125fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311a8c41efe847389bc0c6816e125fff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35405 } }
I20260927 22:14:14.320950 21841 raft_consensus.cc:374] T 00000000000000000000000000000000 P 311a8c41efe847389bc0c6816e125fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:14.321144 21841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 311a8c41efe847389bc0c6816e125fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311a8c41efe847389bc0c6816e125fff, State: Initialized, Role: FOLLOWER
I20260927 22:14:14.321839 21841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 311a8c41efe847389bc0c6816e125fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311a8c41efe847389bc0c6816e125fff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35405 } }
I20260927 22:14:14.322131 21841 raft_consensus.cc:388] T 00000000000000000000000000000000 P 311a8c41efe847389bc0c6816e125fff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:14:14.322276 21841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 311a8c41efe847389bc0c6816e125fff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:14:14.322477 21841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 311a8c41efe847389bc0c6816e125fff [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:14.325573 21841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 311a8c41efe847389bc0c6816e125fff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311a8c41efe847389bc0c6816e125fff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35405 } }
I20260927 22:14:14.326164 21841 leader_election.cc:302] T 00000000000000000000000000000000 P 311a8c41efe847389bc0c6816e125fff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 311a8c41efe847389bc0c6816e125fff; no voters:
I20260927 22:14:14.326964 21841 leader_election.cc:288] T 00000000000000000000000000000000 P 311a8c41efe847389bc0c6816e125fff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:14:14.327065 21851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 311a8c41efe847389bc0c6816e125fff [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:14.329730 21841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 311a8c41efe847389bc0c6816e125fff [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:14:14.330911 21851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 311a8c41efe847389bc0c6816e125fff [term 1 LEADER]: Becoming Leader. State: Replica: 311a8c41efe847389bc0c6816e125fff, State: Running, Role: LEADER
I20260927 22:14:14.331692 21851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 311a8c41efe847389bc0c6816e125fff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311a8c41efe847389bc0c6816e125fff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35405 } }
I20260927 22:14:14.335307 21853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 311a8c41efe847389bc0c6816e125fff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "311a8c41efe847389bc0c6816e125fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311a8c41efe847389bc0c6816e125fff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35405 } } }
I20260927 22:14:14.335671 21853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 311a8c41efe847389bc0c6816e125fff [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:14.336228 21852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 311a8c41efe847389bc0c6816e125fff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 311a8c41efe847389bc0c6816e125fff. Latest consensus state: current_term: 1 leader_uuid: "311a8c41efe847389bc0c6816e125fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311a8c41efe847389bc0c6816e125fff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35405 } } }
I20260927 22:14:14.336534 21852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 311a8c41efe847389bc0c6816e125fff [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:14.336957 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:14.320924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:45569 (local address 127.0.0.1:35405)
0927 22:14:14.321149 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:14.321157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:14.321512 (+ 355us) server_negotiation.cc:407] Connection header received
0927 22:14:14.321684 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:14.321688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:14.321749 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:14.321820 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:14.323683 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:14.324517 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:14.326270 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:14.326879 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:14.327034 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:14.329719 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:14.329724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:14.329727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:14.329758 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:14.333093 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:14.333992 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:14.333997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:14.334000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:14.334175 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:14.334802 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:14.334810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:14.334816 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:14.335338 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:14.335739 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:14.336109 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 22:14:14.336355 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 22:14:14.343433 21859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:14:14.349478 21859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:14:14.358117 21859 catalog_manager.cc:1347] Generated new cluster ID: 6fc6288f184f4db88477052f8c8807de
I20260927 22:14:14.358250 21859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:14:14.372069 21859 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:14:14.373376 21859 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:14:14.382071 21859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 311a8c41efe847389bc0c6816e125fff: Generated new TSK 0
I20260927 22:14:14.382992 21859 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:14:14.393313 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 21774
I20260927 22:14:14.394481 21836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:14:14.486743 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 21774
W20260927 22:14:14.500952 21759 connection.cc:535] client connection to 127.0.0.1:35405 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35405): Connection reset by peer (error 104)
2026-09-27T22:14:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2011 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:14:14 dist-test-slave-mp03 krb5kdc[21873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:14:14 dist-test-slave-mp03 krb5kdc[21873](info): set up 2 sockets
Sep 27 22:14:14 dist-test-slave-mp03 krb5kdc[21873](info): commencing operation
krb5kdc: starting...
W20260927 22:14:15.582991 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:14:15 dist-test-slave-mp03 krb5kdc[21873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:14:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:14:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:14:15.845654 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34965
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:16.174464 21886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:16.174856 21886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:16.175017 21886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:16.182242 21886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:16.182401 21886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:16.182512 21886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:16.182649 21886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:16.182729 21886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:16.192793 21886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34965
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:16.194188 21886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:16.195626 21886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:16.207970 21894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:16.207944 21892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:16.208386 21891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:16.208882 21886 server_base.cc:1034] running on GCE node
I20260927 22:14:16.209672 21886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:16.211200 21886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:16.212435 21886 hybrid_clock.cc:648] HybridClock initialized: now 1790547256212391 us; error 38 us; skew 500 ppm
I20260927 22:14:16.212785 21886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:16 dist-test-slave-mp03 krb5kdc[21873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:16.216185 21886 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:14:16.217604 21886 webserver.cc:460] Webserver started at http://127.0.0.1:37745/ using document root <none> and password file <none>
I20260927 22:14:16.218367 21886 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:16.218513 21886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:16.218845 21886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:16.221843 21886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "143d379206c9417482d428a06a654770"
format_stamp: "Formatted at 2026-09-27 22:14:16 on dist-test-slave-mp03"
server_key: "cbb8c6498262f5b563baee326491a8c4"
server_key_iv: "d32a940b843da96cb4d49438697b4772"
server_key_version: "encryptionkey@0"
I20260927 22:14:16.222640 21886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "143d379206c9417482d428a06a654770"
format_stamp: "Formatted at 2026-09-27 22:14:16 on dist-test-slave-mp03"
server_key: "cbb8c6498262f5b563baee326491a8c4"
server_key_iv: "d32a940b843da96cb4d49438697b4772"
server_key_version: "encryptionkey@0"
I20260927 22:14:16.227773 21886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:14:16.231503 21901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:16.232775 21886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:14:16.232947 21886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "143d379206c9417482d428a06a654770"
format_stamp: "Formatted at 2026-09-27 22:14:16 on dist-test-slave-mp03"
server_key: "cbb8c6498262f5b563baee326491a8c4"
server_key_iv: "d32a940b843da96cb4d49438697b4772"
server_key_version: "encryptionkey@0"
I20260927 22:14:16.233152 21886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:16.250046 21886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:16.254463 21886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:16.254774 21886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:16.272958 21886 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34965
I20260927 22:14:16.272984 21952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34965 every 8 connection(s)
I20260927 22:14:16.274421 21886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:14:16.278147 21953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:16.280313 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 21886
I20260927 22:14:16.280504 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:14:16.280946 388 external_mini_cluster.cc:1428] Setting key e192ec63a848df9f4990c4184ebb82ee
I20260927 22:14:16.289410 21953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 143d379206c9417482d428a06a654770: Bootstrap starting.
I20260927 22:14:16.292392 21953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 143d379206c9417482d428a06a654770: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:16.293521 21953 log.cc:824] T 00000000000000000000000000000000 P 143d379206c9417482d428a06a654770: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:16.293805 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:16.282768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:58093 (local address 127.0.0.1:34965)
0927 22:14:16.283495 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:16.283507 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:16.283548 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:14:16.285157 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:16.285178 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:16.285553 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:16.285921 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:16.286467 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:16.287929 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:16.289260 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:16.289989 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:16.290185 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:16.291454 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:16.291461 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:16.291474 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:14:16.291984 (+ 510us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:16.292053 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:14:16.292076 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:16.292242 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:16.292548 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 22:14:16.292724 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":311,"threads_started":1}
I20260927 22:14:16.296528 21953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 143d379206c9417482d428a06a654770: No bootstrap required, opened a new log
I20260927 22:14:16.297610 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:14:16.305179 21953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 143d379206c9417482d428a06a654770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143d379206c9417482d428a06a654770" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34965 } }
I20260927 22:14:16.305668 21953 raft_consensus.cc:374] T 00000000000000000000000000000000 P 143d379206c9417482d428a06a654770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:16.305780 21953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 143d379206c9417482d428a06a654770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 143d379206c9417482d428a06a654770, State: Initialized, Role: FOLLOWER
I20260927 22:14:16.306448 21953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 143d379206c9417482d428a06a654770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143d379206c9417482d428a06a654770" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34965 } }
I20260927 22:14:16.306807 21953 raft_consensus.cc:388] T 00000000000000000000000000000000 P 143d379206c9417482d428a06a654770 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:14:16.307014 21953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 143d379206c9417482d428a06a654770 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:14:16.307224 21953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 143d379206c9417482d428a06a654770 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:16.308825 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:16.302125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:56547 (local address 127.0.0.1:34965)
0927 22:14:16.302330 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:16.302336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:16.302654 (+ 318us) server_negotiation.cc:407] Connection header received
0927 22:14:16.302837 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:16.302844 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:16.302914 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:16.302992 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:16.303588 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:16.304388 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:16.306222 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:16.306741 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:16.306873 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:16.308023 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:16.308027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:16.308030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:14:16.308047 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:16.308117 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:14:16.308135 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:16.308220 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:16.308419 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 22:14:16.308460 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 22:14:16.309932 21953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 143d379206c9417482d428a06a654770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143d379206c9417482d428a06a654770" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34965 } }
I20260927 22:14:16.310405 21953 leader_election.cc:302] T 00000000000000000000000000000000 P 143d379206c9417482d428a06a654770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 143d379206c9417482d428a06a654770; no voters:
I20260927 22:14:16.311009 21953 leader_election.cc:288] T 00000000000000000000000000000000 P 143d379206c9417482d428a06a654770 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:14:16.311139 21963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 143d379206c9417482d428a06a654770 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:16.311846 21963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 143d379206c9417482d428a06a654770 [term 1 LEADER]: Becoming Leader. State: Replica: 143d379206c9417482d428a06a654770, State: Running, Role: LEADER
I20260927 22:14:16.312409 21963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 143d379206c9417482d428a06a654770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143d379206c9417482d428a06a654770" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34965 } }
I20260927 22:14:16.313088 21953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 143d379206c9417482d428a06a654770 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:14:16.315728 21964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 143d379206c9417482d428a06a654770 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "143d379206c9417482d428a06a654770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143d379206c9417482d428a06a654770" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34965 } } }
I20260927 22:14:16.316087 21964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 143d379206c9417482d428a06a654770 [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:16.316846 21965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 143d379206c9417482d428a06a654770 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 143d379206c9417482d428a06a654770. Latest consensus state: current_term: 1 leader_uuid: "143d379206c9417482d428a06a654770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143d379206c9417482d428a06a654770" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34965 } } }
I20260927 22:14:16.317271 21965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 143d379206c9417482d428a06a654770 [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:16.318076 21970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:14:16.323316 21970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:14:16.333686 21970 catalog_manager.cc:1347] Generated new cluster ID: edd5850fc6ee4c26859b6593167035cf
I20260927 22:14:16.333814 21970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:14:16.367225 21970 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:14:16.369310 21970 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:14:16.377028 21970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 143d379206c9417482d428a06a654770: Generated new TSK 0
I20260927 22:14:16.377897 21970 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:14:16.390699 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 21886
W20260927 22:14:16.391894 21948 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.82:58093
W20260927 22:14:16.392624 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 21886: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:14:16.392757 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 21886
W20260927 22:14:16.407541 21871 connection.cc:535] client connection to 127.0.0.1:34965 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34965): Connection reset by peer (error 104)
2026-09-27T22:14:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1906 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:14:16 dist-test-slave-mp03 krb5kdc[21984](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:14:16 dist-test-slave-mp03 krb5kdc[21984](info): set up 2 sockets
Sep 27 22:14:16 dist-test-slave-mp03 krb5kdc[21984](info): commencing operation
krb5kdc: starting...
W20260927 22:14:17.547094 388 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.012s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:14:17 dist-test-slave-mp03 krb5kdc[21984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:14:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:14:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:14:17.810526 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42491
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:18.150998 21997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:18.151356 21997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:18.151536 21997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:18.159644 21997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:18.159797 21997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:18.159885 21997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:18.159982 21997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:18.160049 21997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:18.170015 21997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42491
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:18.171571 21997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:18.172963 21997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:18.184849 22002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:18.184811 22003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:18.184877 22005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:18.185848 21997 server_base.cc:1034] running on GCE node
I20260927 22:14:18.186714 21997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:18.188200 21997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:18.189409 21997 hybrid_clock.cc:648] HybridClock initialized: now 1790547258189313 us; error 87 us; skew 500 ppm
I20260927 22:14:18.189786 21997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:18 dist-test-slave-mp03 krb5kdc[21984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:18.193017 21997 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:14:18.194586 21997 webserver.cc:460] Webserver started at http://127.0.0.1:34791/ using document root <none> and password file <none>
I20260927 22:14:18.195333 21997 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:18.195461 21997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:18.195781 21997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:18.198843 21997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "258491e5b46247ca9daac27d523ba3a8"
format_stamp: "Formatted at 2026-09-27 22:14:18 on dist-test-slave-mp03"
server_key: "1554999d708737c4eb9ba6e57ab80f99"
server_key_iv: "221df0274ee530a14de8e2c316cf11db"
server_key_version: "encryptionkey@0"
I20260927 22:14:18.199613 21997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "258491e5b46247ca9daac27d523ba3a8"
format_stamp: "Formatted at 2026-09-27 22:14:18 on dist-test-slave-mp03"
server_key: "1554999d708737c4eb9ba6e57ab80f99"
server_key_iv: "221df0274ee530a14de8e2c316cf11db"
server_key_version: "encryptionkey@0"
I20260927 22:14:18.204829 21997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:14:18.208339 22012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:18.209625 21997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:14:18.209847 21997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "258491e5b46247ca9daac27d523ba3a8"
format_stamp: "Formatted at 2026-09-27 22:14:18 on dist-test-slave-mp03"
server_key: "1554999d708737c4eb9ba6e57ab80f99"
server_key_iv: "221df0274ee530a14de8e2c316cf11db"
server_key_version: "encryptionkey@0"
I20260927 22:14:18.210058 21997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:18.215889 21997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:18.220045 21997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:18.220336 21997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:18.239169 21997 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42491
I20260927 22:14:18.239256 22063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42491 every 8 connection(s)
I20260927 22:14:18.240628 21997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:14:18.244393 22064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:18.245139 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 21997
I20260927 22:14:18.245316 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:14:18.245759 388 external_mini_cluster.cc:1428] Setting key 3f7eb3b75aad1deec1b18ccf509225b3
I20260927 22:14:18.255041 22067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:18.247658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:37985 (local address 127.0.0.1:42491)
0927 22:14:18.248431 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:18.248446 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:18.248498 (+ 52us) server_negotiation.cc:407] Connection header received
0927 22:14:18.251190 (+ 2692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:18.251221 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:18.251697 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:18.252161 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:18.253130 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:18.253138 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:18.253155 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:14:18.253170 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:18.253235 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:14:18.253266 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:18.253470 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:18.253597 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 22:14:18.253793 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":282,"threads_started":1}
I20260927 22:14:18.258637 22064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 258491e5b46247ca9daac27d523ba3a8: Bootstrap starting.
I20260927 22:14:18.258680 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:14:18.262616 22064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 258491e5b46247ca9daac27d523ba3a8: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:18.263963 22064 log.cc:824] T 00000000000000000000000000000000 P 258491e5b46247ca9daac27d523ba3a8: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:18.266650 22067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:18.264110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:33605 (local address 127.0.0.1:42491)
0927 22:14:18.264282 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:18.264288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:18.264603 (+ 315us) server_negotiation.cc:407] Connection header received
0927 22:14:18.264804 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:18.264808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:18.264857 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:18.264935 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:18.266000 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:18.266005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:18.266010 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:14:18.266026 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:18.266071 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:14:18.266091 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:18.266226 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:18.266299 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 22:14:18.266359 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:14:18.267290 22064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 258491e5b46247ca9daac27d523ba3a8: No bootstrap required, opened a new log
I20260927 22:14:18.274835 22064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 258491e5b46247ca9daac27d523ba3a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "258491e5b46247ca9daac27d523ba3a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42491 } }
I20260927 22:14:18.275174 22064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 258491e5b46247ca9daac27d523ba3a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:18.275249 22064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 258491e5b46247ca9daac27d523ba3a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 258491e5b46247ca9daac27d523ba3a8, State: Initialized, Role: FOLLOWER
I20260927 22:14:18.275753 22064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 258491e5b46247ca9daac27d523ba3a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "258491e5b46247ca9daac27d523ba3a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42491 } }
I20260927 22:14:18.275955 22064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 258491e5b46247ca9daac27d523ba3a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:14:18.276057 22064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 258491e5b46247ca9daac27d523ba3a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:14:18.276178 22064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 258491e5b46247ca9daac27d523ba3a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:18.278259 22064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 258491e5b46247ca9daac27d523ba3a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "258491e5b46247ca9daac27d523ba3a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42491 } }
I20260927 22:14:18.278842 22064 leader_election.cc:302] T 00000000000000000000000000000000 P 258491e5b46247ca9daac27d523ba3a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 258491e5b46247ca9daac27d523ba3a8; no voters:
I20260927 22:14:18.279459 22064 leader_election.cc:288] T 00000000000000000000000000000000 P 258491e5b46247ca9daac27d523ba3a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:14:18.279655 22074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 258491e5b46247ca9daac27d523ba3a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:18.280475 22074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 258491e5b46247ca9daac27d523ba3a8 [term 1 LEADER]: Becoming Leader. State: Replica: 258491e5b46247ca9daac27d523ba3a8, State: Running, Role: LEADER
I20260927 22:14:18.281224 22074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 258491e5b46247ca9daac27d523ba3a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "258491e5b46247ca9daac27d523ba3a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42491 } }
I20260927 22:14:18.281481 22064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 258491e5b46247ca9daac27d523ba3a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:14:18.285267 22075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 258491e5b46247ca9daac27d523ba3a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "258491e5b46247ca9daac27d523ba3a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "258491e5b46247ca9daac27d523ba3a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42491 } } }
I20260927 22:14:18.285813 22075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 258491e5b46247ca9daac27d523ba3a8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:18.285291 22076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 258491e5b46247ca9daac27d523ba3a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 258491e5b46247ca9daac27d523ba3a8. Latest consensus state: current_term: 1 leader_uuid: "258491e5b46247ca9daac27d523ba3a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "258491e5b46247ca9daac27d523ba3a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42491 } } }
I20260927 22:14:18.286275 22076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 258491e5b46247ca9daac27d523ba3a8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:18.288293 22081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:14:18.294829 22081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:14:18.304425 22081 catalog_manager.cc:1347] Generated new cluster ID: 6f4ce668f62f436a8b8cc1e186639516
I20260927 22:14:18.304571 22081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:14:18.322685 22081 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:14:18.323220 22081 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:14:18.331406 22081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 258491e5b46247ca9daac27d523ba3a8: Generated new TSK 0
I20260927 22:14:18.332444 22081 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:14:18.336720 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 21997
W20260927 22:14:18.337988 22059 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.82:37985
W20260927 22:14:18.338928 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 21997: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:14:18.339071 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 21997
2026-09-27T22:14:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1950 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:14:18 dist-test-slave-mp03 krb5kdc[22095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:14:18 dist-test-slave-mp03 krb5kdc[22095](info): set up 2 sockets
Sep 27 22:14:18 dist-test-slave-mp03 krb5kdc[22095](info): commencing operation
krb5kdc: starting...
W20260927 22:14:19.511562 388 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:14:19 dist-test-slave-mp03 krb5kdc[22095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:14:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:14:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:14:19.770635 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37803
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:20.100036 22108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:20.100330 22108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:20.100497 22108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:20.107769 22108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:20.107887 22108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:20.107976 22108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:20.108119 22108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:20.108171 22108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:20.118476 22108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37803
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:20.119952 22108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:20.121347 22108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:20.133510 22113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:20.133458 22116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:20.133519 22114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:20.134620 22108 server_base.cc:1034] running on GCE node
I20260927 22:14:20.135550 22108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:20.136960 22108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:20.138170 22108 hybrid_clock.cc:648] HybridClock initialized: now 1790547260138126 us; error 35 us; skew 500 ppm
I20260927 22:14:20.138548 22108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:20 dist-test-slave-mp03 krb5kdc[22095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:20.141933 22108 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:14:20.143523 22108 webserver.cc:460] Webserver started at http://127.0.97.62:40943/ using document root <none> and password file <none>
I20260927 22:14:20.144241 22108 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:20.144380 22108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:20.144706 22108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:20.147882 22108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "119a94c29cb34758a5d0af3723c611b8"
format_stamp: "Formatted at 2026-09-27 22:14:20 on dist-test-slave-mp03"
server_key: "9d700c11cfa4017169379355078a7a73"
server_key_iv: "bd5009c4810f7195145002f208fad813"
server_key_version: "encryptionkey@0"
I20260927 22:14:20.148680 22108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "119a94c29cb34758a5d0af3723c611b8"
format_stamp: "Formatted at 2026-09-27 22:14:20 on dist-test-slave-mp03"
server_key: "9d700c11cfa4017169379355078a7a73"
server_key_iv: "bd5009c4810f7195145002f208fad813"
server_key_version: "encryptionkey@0"
I20260927 22:14:20.154126 22108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:14:20.157793 22123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:20.159225 22108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:14:20.159423 22108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "119a94c29cb34758a5d0af3723c611b8"
format_stamp: "Formatted at 2026-09-27 22:14:20 on dist-test-slave-mp03"
server_key: "9d700c11cfa4017169379355078a7a73"
server_key_iv: "bd5009c4810f7195145002f208fad813"
server_key_version: "encryptionkey@0"
I20260927 22:14:20.159637 22108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:20.172644 22108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:20.177011 22108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:20.177305 22108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:20.196355 22108 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37803
I20260927 22:14:20.196378 22174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37803 every 8 connection(s)
I20260927 22:14:20.197979 22108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:14:20.201730 22175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:20.205168 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 22108
I20260927 22:14:20.205348 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:14:20.205822 388 external_mini_cluster.cc:1428] Setting key b75a263be58e2b5b431db97f2da05059
I20260927 22:14:20.214250 22175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 119a94c29cb34758a5d0af3723c611b8: Bootstrap starting.
I20260927 22:14:20.218052 22175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 119a94c29cb34758a5d0af3723c611b8: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:20.219434 22175 log.cc:824] T 00000000000000000000000000000000 P 119a94c29cb34758a5d0af3723c611b8: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:14:20 dist-test-slave-mp03 krb5kdc[22095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:14:20.222477 22175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 119a94c29cb34758a5d0af3723c611b8: No bootstrap required, opened a new log
I20260927 22:14:20.229359 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:20.207656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57613 (local address 127.0.97.62:37803)
0927 22:14:20.208441 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:20.208452 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:20.208491 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:14:20.210120 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:20.210143 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:20.210561 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:20.210972 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:20.212700 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:20.214012 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:20.215827 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:20.216504 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:20.216679 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:20.221877 (+ 5198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:20.221885 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:20.221899 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:20.221936 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:20.225137 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:20.225941 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:20.225947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:20.225953 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:20.226112 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:20.226585 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:20.226588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:20.226633 (+ 45us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:20.227168 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:20.227535 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:20.227992 (+ 457us) server_negotiation.cc:299] Negotiation successful
0927 22:14:20.228281 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":371,"threads_started":1}
I20260927 22:14:20.230515 22175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 119a94c29cb34758a5d0af3723c611b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119a94c29cb34758a5d0af3723c611b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37803 } }
I20260927 22:14:20.230947 22175 raft_consensus.cc:374] T 00000000000000000000000000000000 P 119a94c29cb34758a5d0af3723c611b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:20.231068 22175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 119a94c29cb34758a5d0af3723c611b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 119a94c29cb34758a5d0af3723c611b8, State: Initialized, Role: FOLLOWER
I20260927 22:14:20.231618 22175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 119a94c29cb34758a5d0af3723c611b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119a94c29cb34758a5d0af3723c611b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37803 } }
I20260927 22:14:20.231863 22175 raft_consensus.cc:388] T 00000000000000000000000000000000 P 119a94c29cb34758a5d0af3723c611b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:14:20.232029 22175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 119a94c29cb34758a5d0af3723c611b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:14:20.232210 22175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 119a94c29cb34758a5d0af3723c611b8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:20.233417 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:14:20.234740 22175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 119a94c29cb34758a5d0af3723c611b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119a94c29cb34758a5d0af3723c611b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37803 } }
I20260927 22:14:20.235221 22175 leader_election.cc:302] T 00000000000000000000000000000000 P 119a94c29cb34758a5d0af3723c611b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 119a94c29cb34758a5d0af3723c611b8; no voters:
I20260927 22:14:20.235904 22175 leader_election.cc:288] T 00000000000000000000000000000000 P 119a94c29cb34758a5d0af3723c611b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:14:20.236037 22180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 119a94c29cb34758a5d0af3723c611b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:20.238492 22175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 119a94c29cb34758a5d0af3723c611b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:14:20.242918 22180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 119a94c29cb34758a5d0af3723c611b8 [term 1 LEADER]: Becoming Leader. State: Replica: 119a94c29cb34758a5d0af3723c611b8, State: Running, Role: LEADER
I20260927 22:14:20.243721 22180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 119a94c29cb34758a5d0af3723c611b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119a94c29cb34758a5d0af3723c611b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37803 } }
I20260927 22:14:20.249675 22185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 119a94c29cb34758a5d0af3723c611b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 119a94c29cb34758a5d0af3723c611b8. Latest consensus state: current_term: 1 leader_uuid: "119a94c29cb34758a5d0af3723c611b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119a94c29cb34758a5d0af3723c611b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37803 } } }
I20260927 22:14:20.250155 22185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 119a94c29cb34758a5d0af3723c611b8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:20.250833 22183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 119a94c29cb34758a5d0af3723c611b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "119a94c29cb34758a5d0af3723c611b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "119a94c29cb34758a5d0af3723c611b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37803 } } }
I20260927 22:14:20.254781 22183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 119a94c29cb34758a5d0af3723c611b8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:20.267199 22195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:14:20.271868 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:20.245603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33659 (local address 127.0.97.62:37803)
0927 22:14:20.245850 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:20.245860 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:20.248349 (+ 2489us) server_negotiation.cc:407] Connection header received
0927 22:14:20.248630 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:20.248636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:20.248758 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:20.248850 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:20.251159 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:20.256202 (+ 5043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:20.258936 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:20.260071 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:20.260230 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:20.264586 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:20.264591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:20.264595 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:20.264631 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:20.268494 (+ 3863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:20.269429 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:20.269434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:20.269437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:20.269592 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:20.270127 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:20.270132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:20.270135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:20.270580 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:20.270871 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:20.271332 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 22:14:20.271514 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:14:20.272756 22195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:14:20.281505 22195 catalog_manager.cc:1347] Generated new cluster ID: f65b33bdbf96407ca76860d1643a217e
I20260927 22:14:20.281740 22195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:14:20.301421 22195 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:14:20.303085 22195 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:14:20.312577 22195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 119a94c29cb34758a5d0af3723c611b8: Generated new TSK 0
I20260927 22:14:20.313436 22195 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:14:20.326624 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 22108
I20260927 22:14:20.327847 22170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:14:20.422569 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 22108
W20260927 22:14:20.440183 22093 connection.cc:535] client connection to 127.0.97.62:37803 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:37803): Connection reset by peer (error 104)
2026-09-27T22:14:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2082 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:14:20 dist-test-slave-mp03 krb5kdc[22208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:14:20 dist-test-slave-mp03 krb5kdc[22208](info): set up 2 sockets
Sep 27 22:14:20 dist-test-slave-mp03 krb5kdc[22208](info): commencing operation
krb5kdc: starting...
W20260927 22:14:21.534961 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:14:21 dist-test-slave-mp03 krb5kdc[22208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:14:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:14:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:14:21.794034 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41267
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:22.123780 22221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:22.124066 22221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:22.124286 22221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:22.131580 22221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:22.131701 22221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:22.131817 22221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:22.131918 22221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:22.131996 22221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:22.141924 22221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41267
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:22.143265 22221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:22.144546 22221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:22.156139 22226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:22.156250 22227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:22.156678 22229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:22.157130 22221 server_base.cc:1034] running on GCE node
I20260927 22:14:22.157992 22221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:22.159672 22221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:22.160935 22221 hybrid_clock.cc:648] HybridClock initialized: now 1790547262160898 us; error 59 us; skew 500 ppm
I20260927 22:14:22.161334 22221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:22 dist-test-slave-mp03 krb5kdc[22208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:22.164891 22221 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:14:22.166455 22221 webserver.cc:460] Webserver started at http://127.0.97.62:36887/ using document root <none> and password file <none>
I20260927 22:14:22.167325 22221 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:22.167500 22221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:22.167841 22221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:22.171116 22221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "0d0dccc376c84eed96a86861e932635a"
format_stamp: "Formatted at 2026-09-27 22:14:22 on dist-test-slave-mp03"
server_key: "16ed4658491a1a25f4b65b5a42c692a3"
server_key_iv: "bb701b60fbd3d1ab81548b52cdf08678"
server_key_version: "encryptionkey@0"
I20260927 22:14:22.171958 22221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "0d0dccc376c84eed96a86861e932635a"
format_stamp: "Formatted at 2026-09-27 22:14:22 on dist-test-slave-mp03"
server_key: "16ed4658491a1a25f4b65b5a42c692a3"
server_key_iv: "bb701b60fbd3d1ab81548b52cdf08678"
server_key_version: "encryptionkey@0"
I20260927 22:14:22.177778 22221 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:14:22.181582 22236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:22.183110 22221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:14:22.183306 22221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "0d0dccc376c84eed96a86861e932635a"
format_stamp: "Formatted at 2026-09-27 22:14:22 on dist-test-slave-mp03"
server_key: "16ed4658491a1a25f4b65b5a42c692a3"
server_key_iv: "bb701b60fbd3d1ab81548b52cdf08678"
server_key_version: "encryptionkey@0"
I20260927 22:14:22.183528 22221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:22.209905 22221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:22.214854 22221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:22.215183 22221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:22.235201 22221 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41267
I20260927 22:14:22.235224 22287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41267 every 8 connection(s)
I20260927 22:14:22.236716 22221 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:14:22.238686 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 22221
I20260927 22:14:22.238862 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:14:22.239341 388 external_mini_cluster.cc:1428] Setting key 3cc76c726330300fde9c717068ecb889
I20260927 22:14:22.241338 22288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:22.255303 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:22.241211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56603 (local address 127.0.97.62:41267)
0927 22:14:22.242049 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:22.242065 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:22.242117 (+ 52us) server_negotiation.cc:407] Connection header received
0927 22:14:22.244568 (+ 2451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:22.244599 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:22.245164 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:22.245653 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:22.246182 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:22.248290 (+ 2108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:22.249678 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:22.250830 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:22.251022 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:22.252307 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:22.252325 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:22.252345 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:14:22.253163 (+ 818us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:22.253275 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:14:22.253311 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:22.253490 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:22.253798 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 22:14:22.253993 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":343,"threads_started":1}
I20260927 22:14:22.256488 22288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d0dccc376c84eed96a86861e932635a: Bootstrap starting.
I20260927 22:14:22.259653 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:14:22.260434 22288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d0dccc376c84eed96a86861e932635a: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:22.261488 22288 log.cc:824] T 00000000000000000000000000000000 P 0d0dccc376c84eed96a86861e932635a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:22.264810 22288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d0dccc376c84eed96a86861e932635a: No bootstrap required, opened a new log
I20260927 22:14:22.271585 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:22.264067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47993 (local address 127.0.97.62:41267)
0927 22:14:22.264286 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:22.264295 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:22.264830 (+ 535us) server_negotiation.cc:407] Connection header received
0927 22:14:22.265044 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:22.265048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:22.265106 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:22.265186 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:22.265848 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:22.266586 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:22.268840 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:22.269329 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:22.269434 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:22.270765 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:22.270769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:22.270772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:14:22.270789 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:22.270838 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:14:22.270855 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:22.270950 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:22.271185 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 22:14:22.271231 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:14:22.275352 22288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d0dccc376c84eed96a86861e932635a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d0dccc376c84eed96a86861e932635a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41267 } }
I20260927 22:14:22.275779 22288 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d0dccc376c84eed96a86861e932635a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:22.275893 22288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d0dccc376c84eed96a86861e932635a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d0dccc376c84eed96a86861e932635a, State: Initialized, Role: FOLLOWER
I20260927 22:14:22.276487 22288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d0dccc376c84eed96a86861e932635a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d0dccc376c84eed96a86861e932635a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41267 } }
I20260927 22:14:22.276754 22288 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d0dccc376c84eed96a86861e932635a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:14:22.276906 22288 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d0dccc376c84eed96a86861e932635a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:14:22.277088 22288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d0dccc376c84eed96a86861e932635a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:22.279461 22288 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d0dccc376c84eed96a86861e932635a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d0dccc376c84eed96a86861e932635a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41267 } }
I20260927 22:14:22.279975 22288 leader_election.cc:302] T 00000000000000000000000000000000 P 0d0dccc376c84eed96a86861e932635a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d0dccc376c84eed96a86861e932635a; no voters:
I20260927 22:14:22.280764 22288 leader_election.cc:288] T 00000000000000000000000000000000 P 0d0dccc376c84eed96a86861e932635a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:14:22.280874 22298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d0dccc376c84eed96a86861e932635a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:22.281663 22298 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d0dccc376c84eed96a86861e932635a [term 1 LEADER]: Becoming Leader. State: Replica: 0d0dccc376c84eed96a86861e932635a, State: Running, Role: LEADER
I20260927 22:14:22.282332 22298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d0dccc376c84eed96a86861e932635a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d0dccc376c84eed96a86861e932635a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41267 } }
I20260927 22:14:22.283344 22288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d0dccc376c84eed96a86861e932635a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:14:22.285341 22299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d0dccc376c84eed96a86861e932635a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d0dccc376c84eed96a86861e932635a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d0dccc376c84eed96a86861e932635a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41267 } } }
I20260927 22:14:22.285775 22299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d0dccc376c84eed96a86861e932635a [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:22.286123 22300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d0dccc376c84eed96a86861e932635a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d0dccc376c84eed96a86861e932635a. Latest consensus state: current_term: 1 leader_uuid: "0d0dccc376c84eed96a86861e932635a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d0dccc376c84eed96a86861e932635a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41267 } } }
I20260927 22:14:22.286463 22300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d0dccc376c84eed96a86861e932635a [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:22.287637 22305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:14:22.293093 22305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:14:22.309393 22305 catalog_manager.cc:1347] Generated new cluster ID: 4296e86e46f641f4b7fa27cd3c1be48d
I20260927 22:14:22.309528 22305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:14:22.329726 22305 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:14:22.331552 22305 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:14:22.341637 22305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d0dccc376c84eed96a86861e932635a: Generated new TSK 0
I20260927 22:14:22.342741 22305 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:14:22.361511 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 22221
W20260927 22:14:22.362869 22283 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56603
W20260927 22:14:22.363590 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 22221: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:14:22.363758 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 22221
W20260927 22:14:22.377524 22206 connection.cc:535] client connection to 127.0.97.62:41267 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:41267): Connection reset by peer (error 104)
2026-09-27T22:14:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1936 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:14:22 dist-test-slave-mp03 krb5kdc[22319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:14:22 dist-test-slave-mp03 krb5kdc[22319](info): set up 2 sockets
Sep 27 22:14:22 dist-test-slave-mp03 krb5kdc[22319](info): commencing operation
krb5kdc: starting...
W20260927 22:14:23.478972 388 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:14:23 dist-test-slave-mp03 krb5kdc[22319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:14:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:14:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:14:23.734038 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45895
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:24.064217 22332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:24.064517 22332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:24.064680 22332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:24.072134 22332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:24.072255 22332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:24.072336 22332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:24.072427 22332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:24.072520 22332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:24.082814 22332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45895
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:24.084139 22332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:24.085503 22332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:24.097183 22337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:24.097265 22338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:24.097673 22340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:24.097963 22332 server_base.cc:1034] running on GCE node
I20260927 22:14:24.098825 22332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:24.100593 22332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:24.101804 22332 hybrid_clock.cc:648] HybridClock initialized: now 1790547264101758 us; error 39 us; skew 500 ppm
I20260927 22:14:24.102114 22332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:24 dist-test-slave-mp03 krb5kdc[22319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:24.112807 22332 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:14:24.114418 22332 webserver.cc:460] Webserver started at http://127.0.97.62:46827/ using document root <none> and password file <none>
I20260927 22:14:24.115180 22332 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:24.115336 22332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:24.115633 22332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:24.118572 22332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "cbaf6a11f7244caf9d59663b98661fea"
format_stamp: "Formatted at 2026-09-27 22:14:24 on dist-test-slave-mp03"
server_key: "8fd7d540f1aab24ce52c0c8a7c352684"
server_key_iv: "944895fd57d4ab3c60fb7a8c9de5eecb"
server_key_version: "encryptionkey@0"
I20260927 22:14:24.119413 22332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "cbaf6a11f7244caf9d59663b98661fea"
format_stamp: "Formatted at 2026-09-27 22:14:24 on dist-test-slave-mp03"
server_key: "8fd7d540f1aab24ce52c0c8a7c352684"
server_key_iv: "944895fd57d4ab3c60fb7a8c9de5eecb"
server_key_version: "encryptionkey@0"
I20260927 22:14:24.124449 22332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:14:24.127940 22347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:24.129253 22332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:14:24.129437 22332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "cbaf6a11f7244caf9d59663b98661fea"
format_stamp: "Formatted at 2026-09-27 22:14:24 on dist-test-slave-mp03"
server_key: "8fd7d540f1aab24ce52c0c8a7c352684"
server_key_iv: "944895fd57d4ab3c60fb7a8c9de5eecb"
server_key_version: "encryptionkey@0"
I20260927 22:14:24.129631 22332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:24.135417 22332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:24.139446 22332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:24.139730 22332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:24.158314 22332 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45895
I20260927 22:14:24.158366 22398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45895 every 8 connection(s)
I20260927 22:14:24.159853 22332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:14:24.163894 22399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:24.169121 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 22332
I20260927 22:14:24.169255 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:14:24.169677 388 external_mini_cluster.cc:1428] Setting key a5fdff6adb809866cf0626a0561f0cae
I20260927 22:14:24.176046 22399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbaf6a11f7244caf9d59663b98661fea: Bootstrap starting.
I20260927 22:14:24.178783 22402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:24.171525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37127 (local address 127.0.97.62:45895)
0927 22:14:24.172343 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:24.172360 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:24.172420 (+ 60us) server_negotiation.cc:407] Connection header received
0927 22:14:24.174896 (+ 2476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:24.174924 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:24.175363 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:24.175805 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:24.176801 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:24.176808 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:24.176824 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:14:24.176841 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:24.176914 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:14:24.176942 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:24.177148 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:24.177280 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 22:14:24.177473 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":319,"threads_started":1}
I20260927 22:14:24.179922 22399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbaf6a11f7244caf9d59663b98661fea: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:24.181128 22399 log.cc:824] T 00000000000000000000000000000000 P cbaf6a11f7244caf9d59663b98661fea: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:24.182818 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:14:24.184329 22399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbaf6a11f7244caf9d59663b98661fea: No bootstrap required, opened a new log
I20260927 22:14:24.189941 22402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:24.187314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48303 (local address 127.0.97.62:45895)
0927 22:14:24.187544 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:24.187551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:24.187838 (+ 287us) server_negotiation.cc:407] Connection header received
0927 22:14:24.187983 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:24.187988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:24.188064 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:24.188190 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:24.189180 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:24.189186 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:24.189191 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:14:24.189216 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:24.189273 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:14:24.189295 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:24.189427 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:24.189497 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 22:14:24.189572 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 22:14:24.194010 22399 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbaf6a11f7244caf9d59663b98661fea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbaf6a11f7244caf9d59663b98661fea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45895 } }
I20260927 22:14:24.194386 22399 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbaf6a11f7244caf9d59663b98661fea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:24.194466 22399 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbaf6a11f7244caf9d59663b98661fea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbaf6a11f7244caf9d59663b98661fea, State: Initialized, Role: FOLLOWER
I20260927 22:14:24.195225 22399 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbaf6a11f7244caf9d59663b98661fea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbaf6a11f7244caf9d59663b98661fea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45895 } }
I20260927 22:14:24.195559 22399 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbaf6a11f7244caf9d59663b98661fea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:14:24.195775 22399 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbaf6a11f7244caf9d59663b98661fea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:14:24.196014 22399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbaf6a11f7244caf9d59663b98661fea [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:24.198467 22399 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbaf6a11f7244caf9d59663b98661fea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbaf6a11f7244caf9d59663b98661fea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45895 } }
I20260927 22:14:24.199029 22399 leader_election.cc:302] T 00000000000000000000000000000000 P cbaf6a11f7244caf9d59663b98661fea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbaf6a11f7244caf9d59663b98661fea; no voters:
I20260927 22:14:24.199626 22399 leader_election.cc:288] T 00000000000000000000000000000000 P cbaf6a11f7244caf9d59663b98661fea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:14:24.199734 22409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbaf6a11f7244caf9d59663b98661fea [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:24.200500 22409 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbaf6a11f7244caf9d59663b98661fea [term 1 LEADER]: Becoming Leader. State: Replica: cbaf6a11f7244caf9d59663b98661fea, State: Running, Role: LEADER
I20260927 22:14:24.201043 22409 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbaf6a11f7244caf9d59663b98661fea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbaf6a11f7244caf9d59663b98661fea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45895 } }
I20260927 22:14:24.201925 22399 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbaf6a11f7244caf9d59663b98661fea [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:14:24.203866 22411 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbaf6a11f7244caf9d59663b98661fea [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbaf6a11f7244caf9d59663b98661fea. Latest consensus state: current_term: 1 leader_uuid: "cbaf6a11f7244caf9d59663b98661fea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbaf6a11f7244caf9d59663b98661fea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45895 } } }
I20260927 22:14:24.204391 22411 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbaf6a11f7244caf9d59663b98661fea [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:24.204344 22410 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbaf6a11f7244caf9d59663b98661fea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbaf6a11f7244caf9d59663b98661fea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbaf6a11f7244caf9d59663b98661fea" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45895 } } }
I20260927 22:14:24.204680 22410 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbaf6a11f7244caf9d59663b98661fea [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:24.205605 22416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:14:24.212231 22416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:14:24.221310 22416 catalog_manager.cc:1347] Generated new cluster ID: e4f66b89c41c4319a9a23cf4e8fbe9d2
I20260927 22:14:24.221443 22416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:14:24.237818 22416 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:14:24.238236 22416 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:14:24.252245 22416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbaf6a11f7244caf9d59663b98661fea: Generated new TSK 0
I20260927 22:14:24.253114 22416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:14:24.261447 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 22332
W20260927 22:14:24.262548 22394 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37127
W20260927 22:14:24.263356 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 22332: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:14:24.263497 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 22332
2026-09-27T22:14:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1903 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:14:24 dist-test-slave-mp03 krb5kdc[22430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:14:24 dist-test-slave-mp03 krb5kdc[22430](info): set up 2 sockets
Sep 27 22:14:24 dist-test-slave-mp03 krb5kdc[22430](info): commencing operation
krb5kdc: starting...
W20260927 22:14:25.371217 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:14:25 dist-test-slave-mp03 krb5kdc[22430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:14:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:14:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:14:25.633870 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43829
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:25.973778 22443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:25.974145 22443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:25.974305 22443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:25.981647 22443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:25.981809 22443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:25.981916 22443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:25.982033 22443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:25.982118 22443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:25.992347 22443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:25.993757 22443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:25.995157 22443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:26.007540 22448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:26.007597 22451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:26.007584 22449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:26.008630 22443 server_base.cc:1034] running on GCE node
I20260927 22:14:26.009398 22443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:26.010875 22443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:26.012147 22443 hybrid_clock.cc:648] HybridClock initialized: now 1790547266012076 us; error 52 us; skew 500 ppm
I20260927 22:14:26.012611 22443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:26 dist-test-slave-mp03 krb5kdc[22430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:26.015969 22443 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:14:26.017544 22443 webserver.cc:460] Webserver started at http://127.0.97.62:38129/ using document root <none> and password file <none>
I20260927 22:14:26.018255 22443 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:26.018407 22443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:26.018776 22443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:26.021814 22443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "06e0207042e24bb0b1f5a15f0811c2eb"
format_stamp: "Formatted at 2026-09-27 22:14:26 on dist-test-slave-mp03"
server_key: "b866c4512a854d0d8741560f40a0070d"
server_key_iv: "951d1e600a879915d9b1baff47c0d97c"
server_key_version: "encryptionkey@0"
I20260927 22:14:26.022650 22443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "06e0207042e24bb0b1f5a15f0811c2eb"
format_stamp: "Formatted at 2026-09-27 22:14:26 on dist-test-slave-mp03"
server_key: "b866c4512a854d0d8741560f40a0070d"
server_key_iv: "951d1e600a879915d9b1baff47c0d97c"
server_key_version: "encryptionkey@0"
I20260927 22:14:26.027817 22443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:14:26.031347 22458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:26.032720 22443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:14:26.032934 22443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "06e0207042e24bb0b1f5a15f0811c2eb"
format_stamp: "Formatted at 2026-09-27 22:14:26 on dist-test-slave-mp03"
server_key: "b866c4512a854d0d8741560f40a0070d"
server_key_iv: "951d1e600a879915d9b1baff47c0d97c"
server_key_version: "encryptionkey@0"
I20260927 22:14:26.033138 22443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:26.049675 22443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:26.053745 22443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:26.054049 22443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:26.072705 22443 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43829
I20260927 22:14:26.072729 22509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43829 every 8 connection(s)
I20260927 22:14:26.074167 22443 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:14:26.078002 22510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:26.080035 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 22443
I20260927 22:14:26.080219 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:14:26.080745 388 external_mini_cluster.cc:1428] Setting key 924cee7b00af6727ad6b7c256a8a2d27
I20260927 22:14:26.090971 22510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06e0207042e24bb0b1f5a15f0811c2eb: Bootstrap starting.
I20260927 22:14:26.094177 22510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06e0207042e24bb0b1f5a15f0811c2eb: Neither blocks nor log segments found. Creating new log.
Sep 27 22:14:26 dist-test-slave-mp03 krb5kdc[22430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:14:26.095546 22510 log.cc:824] T 00000000000000000000000000000000 P 06e0207042e24bb0b1f5a15f0811c2eb: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:26.098585 22510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06e0207042e24bb0b1f5a15f0811c2eb: No bootstrap required, opened a new log
I20260927 22:14:26.103581 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:26.082635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37239 (local address 127.0.97.62:43829)
0927 22:14:26.083424 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:26.083436 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:26.083478 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:14:26.085049 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:26.085070 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:26.085500 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:26.085880 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:26.087392 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:26.088717 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:26.090113 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:26.090866 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:26.091068 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:26.095576 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:26.095584 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:26.095599 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:26.095673 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:26.099076 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:26.099928 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:26.099935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:26.099941 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:26.100106 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:26.100542 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:26.100548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:26.100550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:26.101076 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:26.101316 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:26.101766 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 22:14:26.102162 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":354,"threads_started":1}
I20260927 22:14:26.107291 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:14:26.108124 22510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06e0207042e24bb0b1f5a15f0811c2eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e0207042e24bb0b1f5a15f0811c2eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43829 } }
I20260927 22:14:26.108546 22510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06e0207042e24bb0b1f5a15f0811c2eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:26.108634 22510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06e0207042e24bb0b1f5a15f0811c2eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e0207042e24bb0b1f5a15f0811c2eb, State: Initialized, Role: FOLLOWER
I20260927 22:14:26.109253 22510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06e0207042e24bb0b1f5a15f0811c2eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e0207042e24bb0b1f5a15f0811c2eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43829 } }
I20260927 22:14:26.109510 22510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06e0207042e24bb0b1f5a15f0811c2eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:14:26.109647 22510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06e0207042e24bb0b1f5a15f0811c2eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:14:26.109828 22510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06e0207042e24bb0b1f5a15f0811c2eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:26.112988 22510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06e0207042e24bb0b1f5a15f0811c2eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e0207042e24bb0b1f5a15f0811c2eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43829 } }
I20260927 22:14:26.113612 22510 leader_election.cc:302] T 00000000000000000000000000000000 P 06e0207042e24bb0b1f5a15f0811c2eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06e0207042e24bb0b1f5a15f0811c2eb; no voters:
I20260927 22:14:26.114274 22510 leader_election.cc:288] T 00000000000000000000000000000000 P 06e0207042e24bb0b1f5a15f0811c2eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:14:26.114391 22520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06e0207042e24bb0b1f5a15f0811c2eb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:26.116916 22510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06e0207042e24bb0b1f5a15f0811c2eb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:14:26.121511 22520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06e0207042e24bb0b1f5a15f0811c2eb [term 1 LEADER]: Becoming Leader. State: Replica: 06e0207042e24bb0b1f5a15f0811c2eb, State: Running, Role: LEADER
I20260927 22:14:26.122339 22520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06e0207042e24bb0b1f5a15f0811c2eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e0207042e24bb0b1f5a15f0811c2eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43829 } }
I20260927 22:14:26.125823 22521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06e0207042e24bb0b1f5a15f0811c2eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06e0207042e24bb0b1f5a15f0811c2eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e0207042e24bb0b1f5a15f0811c2eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43829 } } }
I20260927 22:14:26.126286 22521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06e0207042e24bb0b1f5a15f0811c2eb [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:26.128429 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:26.112500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45403 (local address 127.0.97.62:43829)
0927 22:14:26.112680 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:26.112686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:26.113110 (+ 424us) server_negotiation.cc:407] Connection header received
0927 22:14:26.113424 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:26.113429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:26.113506 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:26.113619 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:26.115344 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:26.116085 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:26.117485 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:26.117972 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:26.118114 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:26.120585 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:26.120598 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:26.120602 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:26.120631 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:26.123875 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:26.124680 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:26.124685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:26.124688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:26.124860 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:26.125371 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:26.125375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:26.125378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:26.125830 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:26.126072 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:26.127617 (+ 1545us) server_negotiation.cc:299] Negotiation successful
0927 22:14:26.127914 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:14:26.132364 22522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06e0207042e24bb0b1f5a15f0811c2eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06e0207042e24bb0b1f5a15f0811c2eb. Latest consensus state: current_term: 1 leader_uuid: "06e0207042e24bb0b1f5a15f0811c2eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e0207042e24bb0b1f5a15f0811c2eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43829 } } }
I20260927 22:14:26.132742 22522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06e0207042e24bb0b1f5a15f0811c2eb [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:26.137588 22527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:14:26.143627 22527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:14:26.153038 22527 catalog_manager.cc:1347] Generated new cluster ID: df8a324019354cb390d805d5bb42f109
I20260927 22:14:26.153151 22527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:14:26.165141 22527 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:14:26.167070 22527 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:14:26.177682 22527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06e0207042e24bb0b1f5a15f0811c2eb: Generated new TSK 0
I20260927 22:14:26.178794 22527 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:14:26.192857 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 22443
I20260927 22:14:26.194275 22505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:14:26.286787 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 22443
2026-09-27T22:14:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2022 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:14:26 dist-test-slave-mp03 krb5kdc[22543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:14:26 dist-test-slave-mp03 krb5kdc[22543](info): set up 2 sockets
Sep 27 22:14:26 dist-test-slave-mp03 krb5kdc[22543](info): commencing operation
krb5kdc: starting...
W20260927 22:14:27.395170 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:14:27 dist-test-slave-mp03 krb5kdc[22543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:14:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:14:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:14:27.669986 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44965
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:28.004142 22556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:28.004431 22556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:28.004532 22556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:28.011732 22556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:28.011861 22556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:28.011914 22556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:28.011976 22556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:28.012019 22556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:28.021924 22556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:28.023211 22556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:28.024552 22556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:28.036227 22562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:28.036453 22564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:28.036743 22561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:28.037129 22556 server_base.cc:1034] running on GCE node
I20260927 22:14:28.038012 22556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:28.039506 22556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:28.040732 22556 hybrid_clock.cc:648] HybridClock initialized: now 1790547268040690 us; error 29 us; skew 500 ppm
I20260927 22:14:28.041141 22556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:28 dist-test-slave-mp03 krb5kdc[22543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:28.044378 22556 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:14:28.045753 22556 webserver.cc:460] Webserver started at http://127.0.97.62:34417/ using document root <none> and password file <none>
I20260927 22:14:28.046392 22556 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:28.046494 22556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:28.046806 22556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:28.049700 22556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "cfea1b767393498ab782b781bb532653"
format_stamp: "Formatted at 2026-09-27 22:14:28 on dist-test-slave-mp03"
server_key: "547ff17c19a49af7901262fbd5eefb88"
server_key_iv: "0bb8fdeb4524a9d85ce0d73e5a3278cf"
server_key_version: "encryptionkey@0"
I20260927 22:14:28.050400 22556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "cfea1b767393498ab782b781bb532653"
format_stamp: "Formatted at 2026-09-27 22:14:28 on dist-test-slave-mp03"
server_key: "547ff17c19a49af7901262fbd5eefb88"
server_key_iv: "0bb8fdeb4524a9d85ce0d73e5a3278cf"
server_key_version: "encryptionkey@0"
I20260927 22:14:28.055636 22556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:14:28.059329 22571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:28.060716 22556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 22:14:28.060910 22556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "cfea1b767393498ab782b781bb532653"
format_stamp: "Formatted at 2026-09-27 22:14:28 on dist-test-slave-mp03"
server_key: "547ff17c19a49af7901262fbd5eefb88"
server_key_iv: "0bb8fdeb4524a9d85ce0d73e5a3278cf"
server_key_version: "encryptionkey@0"
I20260927 22:14:28.061124 22556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:28.078111 22556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:28.082757 22556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:28.083043 22556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:28.101487 22556 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44965
I20260927 22:14:28.101524 22622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44965 every 8 connection(s)
I20260927 22:14:28.102839 22556 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:14:28.105513 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 22556
I20260927 22:14:28.105700 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:14:28.106245 388 external_mini_cluster.cc:1428] Setting key 7e55db56338eb0ddba3848d1ffc4d1a2
I20260927 22:14:28.106928 22623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:28.119372 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:28.108063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46025 (local address 127.0.97.62:44965)
0927 22:14:28.108820 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:28.108832 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:28.108870 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:14:28.110376 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:28.110397 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:28.110866 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:28.111199 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:28.111764 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:28.113072 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:28.114959 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:28.115593 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:28.115730 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:28.116997 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:28.117005 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:28.117021 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:14:28.117576 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:28.117651 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:14:28.117677 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:28.117764 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:28.118004 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 22:14:28.118315 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":363,"threads_started":1}
I20260927 22:14:28.120489 22623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfea1b767393498ab782b781bb532653: Bootstrap starting.
I20260927 22:14:28.123190 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:14:28.123839 22623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfea1b767393498ab782b781bb532653: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:28.125034 22623 log.cc:824] T 00000000000000000000000000000000 P cfea1b767393498ab782b781bb532653: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:28.128509 22623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfea1b767393498ab782b781bb532653: No bootstrap required, opened a new log
I20260927 22:14:28.135538 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:28.128449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60891 (local address 127.0.97.62:44965)
0927 22:14:28.128657 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:28.128665 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:28.129003 (+ 338us) server_negotiation.cc:407] Connection header received
0927 22:14:28.129142 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:28.129148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:28.129243 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:28.129385 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:28.129862 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:28.131180 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:28.132444 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:28.133309 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:28.133481 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:28.134531 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:28.134545 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:28.134552 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:14:28.134577 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:28.134673 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:14:28.134700 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:28.134836 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:28.134947 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 22:14:28.135067 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 22:14:28.139958 22623 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfea1b767393498ab782b781bb532653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfea1b767393498ab782b781bb532653" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44965 } }
I20260927 22:14:28.140594 22623 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfea1b767393498ab782b781bb532653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:28.140764 22623 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfea1b767393498ab782b781bb532653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfea1b767393498ab782b781bb532653, State: Initialized, Role: FOLLOWER
I20260927 22:14:28.141448 22623 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfea1b767393498ab782b781bb532653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfea1b767393498ab782b781bb532653" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44965 } }
I20260927 22:14:28.141727 22623 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfea1b767393498ab782b781bb532653 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:14:28.141912 22623 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfea1b767393498ab782b781bb532653 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:14:28.142130 22623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfea1b767393498ab782b781bb532653 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:28.144845 22623 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfea1b767393498ab782b781bb532653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfea1b767393498ab782b781bb532653" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44965 } }
I20260927 22:14:28.145519 22623 leader_election.cc:302] T 00000000000000000000000000000000 P cfea1b767393498ab782b781bb532653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfea1b767393498ab782b781bb532653; no voters:
I20260927 22:14:28.146246 22623 leader_election.cc:288] T 00000000000000000000000000000000 P cfea1b767393498ab782b781bb532653 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:14:28.146353 22633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfea1b767393498ab782b781bb532653 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:28.147137 22633 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfea1b767393498ab782b781bb532653 [term 1 LEADER]: Becoming Leader. State: Replica: cfea1b767393498ab782b781bb532653, State: Running, Role: LEADER
I20260927 22:14:28.147796 22633 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfea1b767393498ab782b781bb532653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfea1b767393498ab782b781bb532653" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44965 } }
I20260927 22:14:28.148836 22623 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfea1b767393498ab782b781bb532653 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:14:28.150832 22634 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfea1b767393498ab782b781bb532653 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfea1b767393498ab782b781bb532653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfea1b767393498ab782b781bb532653" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44965 } } }
I20260927 22:14:28.151194 22634 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfea1b767393498ab782b781bb532653 [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:28.150907 22635 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfea1b767393498ab782b781bb532653 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfea1b767393498ab782b781bb532653. Latest consensus state: current_term: 1 leader_uuid: "cfea1b767393498ab782b781bb532653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfea1b767393498ab782b781bb532653" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44965 } } }
I20260927 22:14:28.151407 22635 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfea1b767393498ab782b781bb532653 [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:28.152727 22639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:14:28.157930 22639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:14:28.168701 22639 catalog_manager.cc:1347] Generated new cluster ID: 23636a2bec3e42dc845c656af9a3a73b
I20260927 22:14:28.168831 22639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:14:28.185441 22639 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:14:28.187295 22639 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:14:28.210029 22639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfea1b767393498ab782b781bb532653: Generated new TSK 0
I20260927 22:14:28.211154 22639 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:14:28.225687 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 22556
W20260927 22:14:28.226805 22618 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46025
W20260927 22:14:28.227492 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 22556: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:14:28.227639 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 22556
2026-09-27T22:14:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1940 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:14:28 dist-test-slave-mp03 krb5kdc[22654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:14:28 dist-test-slave-mp03 krb5kdc[22654](info): set up 2 sockets
Sep 27 22:14:28 dist-test-slave-mp03 krb5kdc[22654](info): commencing operation
krb5kdc: starting...
W20260927 22:14:29.331068 388 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:14:29 dist-test-slave-mp03 krb5kdc[22654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:14:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:14:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:14:29.590471 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35357
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:29.920811 22667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:29.921187 22667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:29.921344 22667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:29.928769 22667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:29.928933 22667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:29.929034 22667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:29.929140 22667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:29.929216 22667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:29.939205 22667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:29.940537 22667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:29.941866 22667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:29.953555 22675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:29.953615 22673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:29.953639 22672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:29.954327 22667 server_base.cc:1034] running on GCE node
I20260927 22:14:29.955196 22667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:29.956611 22667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:29.957847 22667 hybrid_clock.cc:648] HybridClock initialized: now 1790547269957808 us; error 34 us; skew 500 ppm
I20260927 22:14:29.958218 22667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:29 dist-test-slave-mp03 krb5kdc[22654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:29.966125 22667 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:14:29.967810 22667 webserver.cc:460] Webserver started at http://127.0.97.62:42273/ using document root <none> and password file <none>
I20260927 22:14:29.968561 22667 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:29.968714 22667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:29.969033 22667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:29.972266 22667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "70792fca58be4ce4b79637cb9117eb8c"
format_stamp: "Formatted at 2026-09-27 22:14:29 on dist-test-slave-mp03"
server_key: "0914f01bc80bbec14bd8245894138a3a"
server_key_iv: "29415f9e12b7869645fcec67c2182f39"
server_key_version: "encryptionkey@0"
I20260927 22:14:29.973078 22667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "70792fca58be4ce4b79637cb9117eb8c"
format_stamp: "Formatted at 2026-09-27 22:14:29 on dist-test-slave-mp03"
server_key: "0914f01bc80bbec14bd8245894138a3a"
server_key_iv: "29415f9e12b7869645fcec67c2182f39"
server_key_version: "encryptionkey@0"
I20260927 22:14:29.978374 22667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 22:14:29.981961 22682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:29.983466 22667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:14:29.983654 22667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "70792fca58be4ce4b79637cb9117eb8c"
format_stamp: "Formatted at 2026-09-27 22:14:29 on dist-test-slave-mp03"
server_key: "0914f01bc80bbec14bd8245894138a3a"
server_key_iv: "29415f9e12b7869645fcec67c2182f39"
server_key_version: "encryptionkey@0"
I20260927 22:14:29.983860 22667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:29.989727 22667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:29.993973 22667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:29.994282 22667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:30.014127 22667 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35357
I20260927 22:14:30.014164 22733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35357 every 8 connection(s)
I20260927 22:14:30.015615 22667 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:14:30.019565 22734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:30.025997 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 22667
I20260927 22:14:30.026185 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:14:30.026788 388 external_mini_cluster.cc:1428] Setting key 233eda31e22194eb61f20e72be39a010
I20260927 22:14:30.031207 22734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70792fca58be4ce4b79637cb9117eb8c: Bootstrap starting.
I20260927 22:14:30.034708 22734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70792fca58be4ce4b79637cb9117eb8c: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:30.035040 22737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:30.028811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51821 (local address 127.0.97.62:35357)
0927 22:14:30.029630 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:30.029642 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:30.029683 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:14:30.031461 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:30.031484 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:30.031837 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:30.032227 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:30.033335 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:30.033342 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:30.033355 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:14:30.033369 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:30.033429 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:14:30.033454 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:30.033679 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:30.033808 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 22:14:30.034000 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":365,"threads_started":1}
I20260927 22:14:30.035869 22734 log.cc:824] T 00000000000000000000000000000000 P 70792fca58be4ce4b79637cb9117eb8c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:30.039062 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:14:30.039259 22734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70792fca58be4ce4b79637cb9117eb8c: No bootstrap required, opened a new log
I20260927 22:14:30.046049 22737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:30.043325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38679 (local address 127.0.97.62:35357)
0927 22:14:30.043536 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:30.043542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:30.044072 (+ 530us) server_negotiation.cc:407] Connection header received
0927 22:14:30.044226 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:30.044229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:30.044277 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:30.044360 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:30.045358 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:30.045362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:30.045365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:14:30.045381 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:30.045417 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:14:30.045434 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:30.045535 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:30.045656 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 22:14:30.045701 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:14:30.048178 22734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70792fca58be4ce4b79637cb9117eb8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70792fca58be4ce4b79637cb9117eb8c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35357 } }
I20260927 22:14:30.048570 22734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70792fca58be4ce4b79637cb9117eb8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:30.048684 22734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70792fca58be4ce4b79637cb9117eb8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70792fca58be4ce4b79637cb9117eb8c, State: Initialized, Role: FOLLOWER
I20260927 22:14:30.049292 22734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70792fca58be4ce4b79637cb9117eb8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70792fca58be4ce4b79637cb9117eb8c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35357 } }
I20260927 22:14:30.049551 22734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70792fca58be4ce4b79637cb9117eb8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:14:30.049697 22734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70792fca58be4ce4b79637cb9117eb8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:14:30.049875 22734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70792fca58be4ce4b79637cb9117eb8c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:30.052606 22734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70792fca58be4ce4b79637cb9117eb8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70792fca58be4ce4b79637cb9117eb8c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35357 } }
I20260927 22:14:30.053126 22734 leader_election.cc:302] T 00000000000000000000000000000000 P 70792fca58be4ce4b79637cb9117eb8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70792fca58be4ce4b79637cb9117eb8c; no voters:
I20260927 22:14:30.053712 22734 leader_election.cc:288] T 00000000000000000000000000000000 P 70792fca58be4ce4b79637cb9117eb8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:14:30.053875 22744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70792fca58be4ce4b79637cb9117eb8c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:30.054699 22744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70792fca58be4ce4b79637cb9117eb8c [term 1 LEADER]: Becoming Leader. State: Replica: 70792fca58be4ce4b79637cb9117eb8c, State: Running, Role: LEADER
I20260927 22:14:30.055267 22744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70792fca58be4ce4b79637cb9117eb8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70792fca58be4ce4b79637cb9117eb8c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35357 } }
I20260927 22:14:30.055996 22734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70792fca58be4ce4b79637cb9117eb8c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:14:30.058228 22746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70792fca58be4ce4b79637cb9117eb8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70792fca58be4ce4b79637cb9117eb8c. Latest consensus state: current_term: 1 leader_uuid: "70792fca58be4ce4b79637cb9117eb8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70792fca58be4ce4b79637cb9117eb8c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35357 } } }
I20260927 22:14:30.058575 22746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70792fca58be4ce4b79637cb9117eb8c [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:30.059011 22745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70792fca58be4ce4b79637cb9117eb8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70792fca58be4ce4b79637cb9117eb8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70792fca58be4ce4b79637cb9117eb8c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35357 } } }
I20260927 22:14:30.061172 22745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70792fca58be4ce4b79637cb9117eb8c [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:30.065550 22751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:14:30.071544 22751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:14:30.080590 22751 catalog_manager.cc:1347] Generated new cluster ID: dc5c9d27043949cc9c44d241b850f4ca
I20260927 22:14:30.080719 22751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:14:30.107656 22751 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:14:30.108150 22751 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:14:30.115298 22751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70792fca58be4ce4b79637cb9117eb8c: Generated new TSK 0
I20260927 22:14:30.116147 22751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:14:30.126825 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 22667
W20260927 22:14:30.127975 22729 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51821
W20260927 22:14:30.128656 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 22667: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:14:30.128806 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 22667
2026-09-27T22:14:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1900 ms)
[----------] 18 tests from AuthTokenIssuingTest (35350 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:14:30 dist-test-slave-mp03 krb5kdc[22765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:14:30 dist-test-slave-mp03 krb5kdc[22765](info): set up 2 sockets
Sep 27 22:14:30 dist-test-slave-mp03 krb5kdc[22765](info): commencing operation
krb5kdc: starting...
W20260927 22:14:31.235530 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:14:31 dist-test-slave-mp03 krb5kdc[22765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:14:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:14:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:14:31.498566 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35327
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35327,127.0.97.61:39587,127.0.97.60:40717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:31.830969 22778 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:14:31.831331 22778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:31.831445 22778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:31.831570 22778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:31.838938 22778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:31.839035 22778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:31.839133 22778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:31.839201 22778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:31.839285 22778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:31.849417 22778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35327,127.0.97.61:39587,127.0.97.60:40717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:31.850809 22778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:31.852211 22778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:31.863961 22783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:31.864224 22784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:31.864956 22786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:31.865551 22778 server_base.cc:1034] running on GCE node
I20260927 22:14:31.866386 22778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:31.867962 22778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:31.869199 22778 hybrid_clock.cc:648] HybridClock initialized: now 1790547271869114 us; error 51 us; skew 500 ppm
I20260927 22:14:31.869619 22778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:31 dist-test-slave-mp03 krb5kdc[22765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:31.872947 22778 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:14:31.874473 22778 webserver.cc:460] Webserver started at http://127.0.97.62:42997/ using document root <none> and password file <none>
I20260927 22:14:31.875239 22778 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:31.875329 22778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:31.875697 22778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:31.878831 22778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "ff0f1e2a420845428fe8138c02394161"
format_stamp: "Formatted at 2026-09-27 22:14:31 on dist-test-slave-mp03"
server_key: "d3cf3ebf48169c22c5f60b226cee254a"
server_key_iv: "2a11cd6e101a1c85ce182f197ef22d02"
server_key_version: "encryptionkey@0"
I20260927 22:14:31.879688 22778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "ff0f1e2a420845428fe8138c02394161"
format_stamp: "Formatted at 2026-09-27 22:14:31 on dist-test-slave-mp03"
server_key: "d3cf3ebf48169c22c5f60b226cee254a"
server_key_iv: "2a11cd6e101a1c85ce182f197ef22d02"
server_key_version: "encryptionkey@0"
I20260927 22:14:31.885003 22778 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 22:14:31.888427 22793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:31.889844 22778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:14:31.890038 22778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "ff0f1e2a420845428fe8138c02394161"
format_stamp: "Formatted at 2026-09-27 22:14:31 on dist-test-slave-mp03"
server_key: "d3cf3ebf48169c22c5f60b226cee254a"
server_key_iv: "2a11cd6e101a1c85ce182f197ef22d02"
server_key_version: "encryptionkey@0"
I20260927 22:14:31.890261 22778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:31.918028 22778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:31.922816 22778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:31.923106 22778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:31.942024 22778 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35327
I20260927 22:14:31.942067 22844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35327 every 8 connection(s)
I20260927 22:14:31.943434 22778 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:14:31.944547 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 22778
I20260927 22:14:31.944767 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:14:31.945559 388 external_mini_cluster.cc:1428] Setting key f9e51495623cb608efdc210846c40f60
I20260927 22:14:31.948246 22845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:31.957495 22845 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35327 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 27 22:14:31 dist-test-slave-mp03 krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:14:31.983476 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:31.960707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58170 (local address 127.0.97.62:35327)
0927 22:14:31.962403 (+ 1696us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:31.962411 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:31.962458 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:14:31.964108 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:31.964114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:31.964336 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:31.964536 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:31.965852 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:31.967152 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:31.968617 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:31.969186 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:31.969397 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:31.975151 (+ 5754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:31.975165 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:31.975186 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:31.975234 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:31.979165 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:31.979896 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:31.979908 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:31.979914 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:31.980113 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:31.980498 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:31.980506 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:31.980512 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:31.981238 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:31.981450 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:31.982003 (+ 553us) server_negotiation.cc:299] Negotiation successful
0927 22:14:31.982443 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":101,"server-negotiator.queue_time_us":1473,"thread_start_us":1269,"threads_started":1}
I20260927 22:14:31.983434 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:31.960707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35327 (local address 127.0.0.1:58170)
0927 22:14:31.961595 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:31.961640 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:31.963748 (+ 2108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:31.964561 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:31.964568 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:31.965316 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:31.965647 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:31.965664 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:31.967304 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:31.967310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:31.968472 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:31.968484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:31.969420 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:31.969426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:31.969607 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:31.970270 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:31.970292 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:31.974861 (+ 4569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:31.979318 (+ 4457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:31.979328 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:31.979338 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:31.979758 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:31.980228 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:31.980231 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:31.980233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:31.980385 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:31.981371 (+ 986us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:31.981377 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:31.981607 (+ 230us) client_negotiation.cc:769] Sending connection context
0927 22:14:31.981817 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 22:14:31.982230 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":393,"threads_started":1}
I20260927 22:14:31.986572 22845 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:14:31.987658 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:31.987266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39587 (local address 127.0.0.1:34336)
0927 22:14:31.987435 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:31.987539 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 22:14:31.987987 22795 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39587: connect: Connection refused (error 111)
I20260927 22:14:31.989434 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:31.989106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39587 (local address 127.0.0.1:34344)
0927 22:14:31.989243 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:31.989319 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 22:14:31.989669 22845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39587: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39587: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260927 22:14:32.010551 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39587
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35327,127.0.97.61:39587,127.0.97.60:40717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:14:32.039094 22845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } attempt: 1
I20260927 22:14:32.040369 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.039923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39587 (local address 127.0.0.1:34356)
0927 22:14:32.040152 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.040242 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260927 22:14:32.041420 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.041086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39587 (local address 127.0.0.1:34366)
0927 22:14:32.041237 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.041299 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 22:14:32.041632 22845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39587: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39587: connect: Connection refused (error 111)
I20260927 22:14:32.109871 22845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } attempt: 2
I20260927 22:14:32.111266 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.110707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39587 (local address 127.0.0.1:34376)
0927 22:14:32.111025 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.111115 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 22:14:32.112406 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.112040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39587 (local address 127.0.0.1:34384)
0927 22:14:32.112198 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.112268 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 22:14:32.112593 22845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39587: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39587: connect: Connection refused (error 111)
I20260927 22:14:32.203850 22845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } attempt: 3
I20260927 22:14:32.205044 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.204559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39587 (local address 127.0.0.1:34398)
0927 22:14:32.204768 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.204898 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 22:14:32.206174 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.205790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39587 (local address 127.0.0.1:34410)
0927 22:14:32.205977 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.206041 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 22:14:32.206354 22845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39587: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39587: connect: Connection refused (error 111)
I20260927 22:14:32.349601 22845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } attempt: 4
I20260927 22:14:32.350924 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.350335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39587 (local address 127.0.0.1:34422)
0927 22:14:32.350684 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.350800 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260927 22:14:32.351910 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.351558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39587 (local address 127.0.0.1:34438)
0927 22:14:32.351709 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.351782 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 22:14:32.352140 22845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39587: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39587: connect: Connection refused (error 111)
W20260927 22:14:32.355764 22852 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:14:32.356133 22852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:32.356264 22852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:32.356415 22852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:32.364221 22852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:32.364377 22852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:32.364523 22852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:32.364619 22852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:32.364739 22852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:32.375942 22852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35327,127.0.97.61:39587,127.0.97.60:40717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:32.377336 22852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:32.378834 22852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:32.390875 22857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:32.390990 22858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:32.391050 22860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:32.392167 22852 server_base.cc:1034] running on GCE node
I20260927 22:14:32.392910 22852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:32.394379 22852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:32.395630 22852 hybrid_clock.cc:648] HybridClock initialized: now 1790547272395558 us; error 50 us; skew 500 ppm
I20260927 22:14:32.395951 22852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:32 dist-test-slave-mp03 krb5kdc[22765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:32.399014 22852 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260927 22:14:32.400362 22852 webserver.cc:460] Webserver started at http://127.0.97.61:44171/ using document root <none> and password file <none>
I20260927 22:14:32.401033 22852 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:32.401161 22852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:32.401479 22852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:32.404426 22852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-1/data/instance:
uuid: "9c1803db4add4de8849645932809c607"
format_stamp: "Formatted at 2026-09-27 22:14:32 on dist-test-slave-mp03"
server_key: "24e3cfeda6307ceee6a11d27fa02ec62"
server_key_iv: "f844ecc268399c31682c2872f1f7f079"
server_key_version: "encryptionkey@0"
I20260927 22:14:32.405326 22852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-1/wal/instance:
uuid: "9c1803db4add4de8849645932809c607"
format_stamp: "Formatted at 2026-09-27 22:14:32 on dist-test-slave-mp03"
server_key: "24e3cfeda6307ceee6a11d27fa02ec62"
server_key_iv: "f844ecc268399c31682c2872f1f7f079"
server_key_version: "encryptionkey@0"
I20260927 22:14:32.410846 22852 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 22:14:32.414376 22867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:32.415789 22852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:14:32.416034 22852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-1/wal
uuid: "9c1803db4add4de8849645932809c607"
format_stamp: "Formatted at 2026-09-27 22:14:32 on dist-test-slave-mp03"
server_key: "24e3cfeda6307ceee6a11d27fa02ec62"
server_key_iv: "f844ecc268399c31682c2872f1f7f079"
server_key_version: "encryptionkey@0"
I20260927 22:14:32.416268 22852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:32.431310 22852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:32.435470 22852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:32.435732 22852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:32.454870 22852 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:39587
I20260927 22:14:32.454945 22918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:39587 every 8 connection(s)
I20260927 22:14:32.456269 22852 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-1/data/info.pb
I20260927 22:14:32.459914 22919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:32.466404 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 22852
I20260927 22:14:32.466643 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-1/wal/instance
I20260927 22:14:32.467063 388 external_mini_cluster.cc:1428] Setting key 0ec9e5c78c1a56c4cc8b370dd028c648
I20260927 22:14:32.466310 22919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35327 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:14:32 dist-test-slave-mp03 krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:14:32.488754 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.469094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58174 (local address 127.0.97.62:35327)
0927 22:14:32.469272 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:32.469278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:32.470531 (+ 1253us) server_negotiation.cc:407] Connection header received
0927 22:14:32.472499 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:32.472504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:32.472594 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:32.472705 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:32.474211 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:32.474984 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:32.476859 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:32.477287 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:32.477387 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:32.481525 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:32.481534 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:32.481542 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:32.481589 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:32.485325 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:32.486124 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:32.486133 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:32.486139 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:32.486285 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:32.486741 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:32.486748 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:32.486754 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:32.487152 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:32.487314 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:32.488046 (+ 732us) server_negotiation.cc:299] Negotiation successful
0927 22:14:32.488312 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:14:32.489502 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.469590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35327 (local address 127.0.0.1:58174)
0927 22:14:32.470360 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.470404 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:32.472227 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:32.472936 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:32.472944 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:32.473447 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:32.474036 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:32.474054 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:32.475176 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:32.475181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:32.476690 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:32.476708 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:32.477517 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:32.477526 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:32.477860 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:32.478641 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:32.478676 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:32.481288 (+ 2612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:32.485469 (+ 4181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:32.485476 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:32.485491 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:32.485998 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:32.486391 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:32.486394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:32.486396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:32.486580 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:32.487293 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:32.487299 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:32.487654 (+ 355us) client_negotiation.cc:769] Sending connection context
0927 22:14:32.487888 (+ 234us) client_negotiation.cc:240] Negotiation successful
0927 22:14:32.488289 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":348,"threads_started":1}
I20260927 22:14:32.491561 22919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 27 22:14:32 dist-test-slave-mp03 krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:14:32.509559 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.492610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34454 (local address 127.0.97.61:39587)
0927 22:14:32.493385 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:32.493395 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:32.493436 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:14:32.493497 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:32.493505 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:32.493754 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:32.493968 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:32.494665 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:32.495776 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:32.497170 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:32.498130 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:32.498272 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:32.502513 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:32.502528 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:32.502549 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:32.502633 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:32.506565 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:32.507429 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:32.507435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:32.507437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:32.507558 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:32.507956 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:32.507959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:32.507961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:32.508334 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:32.508492 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:32.508942 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 22:14:32.509268 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":595,"thread_start_us":435,"threads_started":1}
I20260927 22:14:32.509637 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.492286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39587 (local address 127.0.0.1:34454)
0927 22:14:32.492550 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.492571 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:32.492682 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:32.493938 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:32.493944 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:32.494301 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:32.494440 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:32.494448 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:32.495923 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:32.495929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:32.497012 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:32.497024 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:32.498487 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:32.498495 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:32.498732 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:32.499689 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:32.499712 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:32.502288 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:32.506720 (+ 4432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:32.506728 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:32.506733 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:32.507306 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:32.507674 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:32.507682 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:32.507688 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:32.507852 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:32.508463 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:32.508470 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:32.508671 (+ 201us) client_negotiation.cc:769] Sending connection context
0927 22:14:32.508825 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 22:14:32.509151 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 22:14:32.512019 22919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:14:32.513139 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.512632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40717 (local address 127.0.0.1:39316)
0927 22:14:32.512816 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.512953 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 22:14:32.513389 22869 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
I20260927 22:14:32.515019 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.514579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40717 (local address 127.0.0.1:39328)
0927 22:14:32.514792 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.514885 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260927 22:14:32.515249 22919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40717: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260927 22:14:32.531005 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40717
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35327,127.0.97.61:39587,127.0.97.60:40717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:14:32.564599 22919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } attempt: 1
I20260927 22:14:32.565907 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.565324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40717 (local address 127.0.0.1:39334)
0927 22:14:32.565579 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.565697 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 22:14:32.566984 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.566562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40717 (local address 127.0.0.1:39338)
0927 22:14:32.566737 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.566847 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260927 22:14:32.567227 22919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40717: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
I20260927 22:14:32.635486 22919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } attempt: 2
I20260927 22:14:32.636732 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.636146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40717 (local address 127.0.0.1:39352)
0927 22:14:32.636373 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.636510 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 22:14:32.637703 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.637312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40717 (local address 127.0.0.1:39354)
0927 22:14:32.637486 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.637568 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 22:14:32.637933 22919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40717: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
I20260927 22:14:32.651364 22845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } attempt: 5
Sep 27 22:14:32 dist-test-slave-mp03 krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:14:32.666968 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.652086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39587 (local address 127.0.0.1:34468)
0927 22:14:32.652312 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.652334 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:32.652493 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:32.652842 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:32.652847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:32.653345 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:32.653587 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:32.653600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:32.654566 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:32.654571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:32.655915 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:32.655932 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:32.656741 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:32.656747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:32.656923 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:32.657644 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:32.657675 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:32.660534 (+ 2859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:32.664584 (+ 4050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:32.664593 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:32.664598 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:32.665042 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:32.665443 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:32.665447 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:32.665449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:32.665578 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:32.666175 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:32.666178 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:32.666307 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 22:14:32.666419 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 22:14:32.666691 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 22:14:32.667145 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.652151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34468 (local address 127.0.97.61:39587)
0927 22:14:32.652330 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:32.652335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:32.652417 (+ 82us) server_negotiation.cc:407] Connection header received
0927 22:14:32.652624 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:32.652628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:32.652691 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:32.652776 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:32.653733 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:32.654414 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:32.656083 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:32.656573 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:32.656667 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:32.660748 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:32.660756 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:32.660762 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:32.660809 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:32.664376 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:32.665169 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:32.665174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:32.665178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:32.665332 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:32.665681 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:32.665686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:32.665689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:32.666056 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:32.666214 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:32.666570 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 22:14:32.666852 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:14:32.668179 22845 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:14:32.669219 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.668815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40717 (local address 127.0.0.1:39356)
0927 22:14:32.668977 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.669083 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 22:14:32.670096 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.669781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40717 (local address 127.0.0.1:39370)
0927 22:14:32.669922 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.669988 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 22:14:32.670312 22845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40717: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
I20260927 22:14:32.721540 22845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } attempt: 1
I20260927 22:14:32.722761 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.722237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40717 (local address 127.0.0.1:39384)
0927 22:14:32.722434 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.722531 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 22:14:32.723829 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.723482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40717 (local address 127.0.0.1:39388)
0927 22:14:32.723663 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.723730 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 22:14:32.724044 22845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40717: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
I20260927 22:14:32.729178 22919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } attempt: 3
I20260927 22:14:32.730415 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.729824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40717 (local address 127.0.0.1:39392)
0927 22:14:32.730076 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.730197 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 22:14:32.731734 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.731162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40717 (local address 127.0.0.1:39406)
0927 22:14:32.731433 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.731556 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260927 22:14:32.732045 22919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40717: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
I20260927 22:14:32.792253 22845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } attempt: 2
I20260927 22:14:32.793560 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.793050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40717 (local address 127.0.0.1:39412)
0927 22:14:32.793264 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.793375 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 22:14:32.794884 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.794415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40717 (local address 127.0.0.1:39424)
0927 22:14:32.794573 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.794724 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 22:14:32.795126 22845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40717: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
W20260927 22:14:32.873036 22926 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:14:32.873397 22926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:32.873528 22926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:32.873649 22926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 22:14:32.875283 22919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } attempt: 4
I20260927 22:14:32.876492 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.875955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40717 (local address 127.0.0.1:39428)
0927 22:14:32.876199 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.876305 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 22:14:32.877522 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.877155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40717 (local address 127.0.0.1:39436)
0927 22:14:32.877308 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.877389 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 22:14:32.877717 22919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40717: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
W20260927 22:14:32.881910 22926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:32.881992 22926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:32.882136 22926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:32.882215 22926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:32.882325 22926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:32.892493 22926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35327,127.0.97.61:39587,127.0.97.60:40717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:32.893843 22926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:32.895309 22926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 22:14:32.901372 22845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } attempt: 3
I20260927 22:14:32.902460 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.902000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40717 (local address 127.0.0.1:39440)
0927 22:14:32.902221 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.902311 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 22:14:32.903545 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:32.903193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40717 (local address 127.0.0.1:39448)
0927 22:14:32.903346 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:32.903441 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 22:14:32.903807 22845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40717: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40717: connect: Connection refused (error 111)
W20260927 22:14:32.907521 22934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:32.907634 22931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:32.907604 22932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:32.912573 22926 server_base.cc:1034] running on GCE node
I20260927 22:14:32.913455 22926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:32.915103 22926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:32.916332 22926 hybrid_clock.cc:648] HybridClock initialized: now 1790547272916249 us; error 56 us; skew 500 ppm
I20260927 22:14:32.916644 22926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:32 dist-test-slave-mp03 krb5kdc[22765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:32.919824 22926 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260927 22:14:32.921330 22926 webserver.cc:460] Webserver started at http://127.0.97.60:40349/ using document root <none> and password file <none>
I20260927 22:14:32.922096 22926 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:32.922228 22926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:32.922559 22926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:32.925573 22926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-2/data/instance:
uuid: "2cb170e6f8ee478eae8cdb8ab294ec12"
format_stamp: "Formatted at 2026-09-27 22:14:32 on dist-test-slave-mp03"
server_key: "e8c195dca7e72b4cf2d7ff14580dc5ee"
server_key_iv: "7dd3fc128cb502af0c508f4953dfcd25"
server_key_version: "encryptionkey@0"
I20260927 22:14:32.926440 22926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-2/wal/instance:
uuid: "2cb170e6f8ee478eae8cdb8ab294ec12"
format_stamp: "Formatted at 2026-09-27 22:14:32 on dist-test-slave-mp03"
server_key: "e8c195dca7e72b4cf2d7ff14580dc5ee"
server_key_iv: "7dd3fc128cb502af0c508f4953dfcd25"
server_key_version: "encryptionkey@0"
I20260927 22:14:32.931733 22926 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260927 22:14:32.935679 22941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:32.937203 22926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:14:32.937422 22926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-2/wal
uuid: "2cb170e6f8ee478eae8cdb8ab294ec12"
format_stamp: "Formatted at 2026-09-27 22:14:32 on dist-test-slave-mp03"
server_key: "e8c195dca7e72b4cf2d7ff14580dc5ee"
server_key_iv: "7dd3fc128cb502af0c508f4953dfcd25"
server_key_version: "encryptionkey@0"
I20260927 22:14:32.937683 22926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:32.969923 22926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:32.974462 22926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:32.974795 22926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:32.995090 22926 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:40717
I20260927 22:14:32.995103 22992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:40717 every 8 connection(s)
I20260927 22:14:32.996596 22926 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-2/data/info.pb
I20260927 22:14:33.000433 22993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:33.006319 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 22926
I20260927 22:14:33.006510 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546912983837-388-0/minicluster-data/master-2/wal/instance
I20260927 22:14:33.007041 388 external_mini_cluster.cc:1428] Setting key c2ebbff68dcd0166d8fdd53e7227efc4
I20260927 22:14:33.007252 22993 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35327 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:14:33 dist-test-slave-mp03 krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 27 22:14:33 dist-test-slave-mp03 krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 22:14:33.028538 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:33.010268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58178 (local address 127.0.97.62:35327)
0927 22:14:33.011102 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:33.011111 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:33.011308 (+ 197us) server_negotiation.cc:407] Connection header received
0927 22:14:33.012444 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:33.012450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:33.012553 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:33.012668 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:33.013998 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.014821 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.016424 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.016981 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.017074 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:33.021105 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:33.021109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:33.021112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:33.021145 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:33.024975 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:33.025823 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:33.025828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:33.025832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:33.026054 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:33.026462 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:33.026469 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:33.026472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:33.026809 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:33.026982 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:33.027687 (+ 705us) server_negotiation.cc:299] Negotiation successful
0927 22:14:33.027950 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":544,"threads_started":1}
I20260927 22:14:33.029425 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:33.009017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39450 (local address 127.0.97.60:40717)
0927 22:14:33.009773 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:33.009784 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:33.009820 (+ 36us) server_negotiation.cc:407] Connection header received
0927 22:14:33.012174 (+ 2354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:33.012194 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:33.012418 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:33.012576 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:33.013951 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.015346 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.016754 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.017532 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.017671 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:33.022124 (+ 4453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:33.022135 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:33.022150 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:33.022189 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:33.025829 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:33.026432 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:33.026441 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:33.026445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:33.026581 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:33.026965 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:33.026972 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:33.026975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:33.027374 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:33.027679 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:33.028016 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 22:14:33.028339 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":562,"thread_start_us":345,"threads_started":1}
I20260927 22:14:33.029452 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:33.010179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35327 (local address 127.0.0.1:58178)
0927 22:14:33.011171 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:33.011205 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:33.012188 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:33.012768 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:33.012775 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:33.013444 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:33.013800 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:33.013816 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.014962 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.014967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:33.016270 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:33.016282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.017184 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.017191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:33.017564 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:33.018363 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:33.018392 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:33.020909 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:33.025107 (+ 4198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:33.025116 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:33.025125 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:33.025704 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:33.026164 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:33.026172 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:33.026176 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:33.026345 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:33.026935 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:33.026942 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:33.027276 (+ 334us) client_negotiation.cc:769] Sending connection context
0927 22:14:33.027590 (+ 314us) client_negotiation.cc:240] Negotiation successful
0927 22:14:33.028203 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":59,"spinlock_wait_cycles":14080,"thread_start_us":355,"threads_started":1}
I20260927 22:14:33.031730 22993 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:14:33 dist-test-slave-mp03 krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 27 22:14:33 dist-test-slave-mp03 krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:14:33.046166 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:33.032413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39587 (local address 127.0.0.1:34484)
0927 22:14:33.032643 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:33.032660 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:33.032756 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:33.033309 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:33.033313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:33.033657 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:33.033794 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:33.033801 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.034813 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.034818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:33.035669 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:33.035679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.036388 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.036394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:33.036575 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:33.037121 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:33.037136 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:33.039815 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:33.043590 (+ 3775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:33.043594 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:33.043597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:33.043982 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:33.044438 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:33.044442 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:33.044444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:33.044591 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:33.045206 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:33.045210 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:33.045356 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 22:14:33.045504 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 22:14:33.045790 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260927 22:14:33.046195 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:33.032520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34484 (local address 127.0.97.61:39587)
0927 22:14:33.032693 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:33.032699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:33.032717 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:14:33.032907 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:33.032912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:33.033018 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:33.033091 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:33.033944 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.034670 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.035804 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.036246 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.036372 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:33.040075 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:33.040080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:33.040083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:33.040116 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:33.043452 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:33.044147 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:33.044153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:33.044157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:33.044324 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:33.044698 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:33.044706 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:33.044712 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:33.045090 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:33.045239 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:33.045594 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 22:14:33.045797 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:14:33.047065 22993 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:14:33.047936 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:33.032636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58194 (local address 127.0.97.62:35327)
0927 22:14:33.032880 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:33.032885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:33.033002 (+ 117us) server_negotiation.cc:407] Connection header received
0927 22:14:33.033099 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:33.033103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:33.033184 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:33.033279 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:33.034362 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.035112 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.036282 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.036699 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.036843 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:33.041518 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:33.041526 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:33.041533 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:33.041575 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:33.044860 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:33.045540 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:33.045545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:33.045547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:33.045668 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:33.046025 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:33.046032 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:33.046037 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:33.046398 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:33.046692 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:33.047162 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 22:14:33.047390 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
Sep 27 22:14:33 dist-test-slave-mp03 krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 22:14:33 dist-test-slave-mp03 krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:14:33.061504 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:33.047970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39458 (local address 127.0.97.60:40717)
0927 22:14:33.048138 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:33.048144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:33.048159 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:14:33.048205 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:33.048209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:33.048272 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:33.048378 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:33.049228 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.050186 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.051918 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.052567 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.052692 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:33.056232 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:33.056238 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:33.056243 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:33.056279 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:33.059051 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:33.059631 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:33.059638 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:33.059641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:33.059780 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:33.060144 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:33.060151 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:33.060154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:33.060422 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:33.060566 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:33.060902 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 22:14:33.061141 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:14:33.061806 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:33.047767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40717 (local address 127.0.0.1:39458)
0927 22:14:33.047952 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:33.047969 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:33.048076 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:33.048422 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:33.048426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:33.048913 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:33.049093 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:33.049102 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.050313 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.050318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:33.051788 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:33.051799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.052714 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.052719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:33.052894 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:33.053473 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:33.053496 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:33.056048 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:33.059175 (+ 3127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:33.059181 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:33.059185 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:33.059520 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:33.059893 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:33.059897 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:33.059901 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:33.060042 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:33.060528 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:33.060532 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:33.060671 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 22:14:33.060823 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 22:14:33.061331 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":184}
I20260927 22:14:33.065060 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:33.049673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34490 (local address 127.0.97.61:39587)
0927 22:14:33.049829 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:33.049834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:33.049852 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:14:33.049951 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:33.049956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:33.050022 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:33.050110 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:33.051729 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.052789 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.054562 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.055008 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.055152 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:33.059589 (+ 4437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:33.059594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:33.059597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:33.059628 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:33.062507 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:33.063161 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:33.063168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:33.063172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:33.063305 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:33.063717 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:33.063722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:33.063725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:33.063983 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:33.064257 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:33.064529 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 22:14:33.064740 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:14:33.067032 22993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cb170e6f8ee478eae8cdb8ab294ec12: Bootstrap starting.
I20260927 22:14:33.069789 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:14:33.070534 22993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cb170e6f8ee478eae8cdb8ab294ec12: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:33.071580 22993 log.cc:824] T 00000000000000000000000000000000 P 2cb170e6f8ee478eae8cdb8ab294ec12: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:33.074263 22993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cb170e6f8ee478eae8cdb8ab294ec12: No bootstrap required, opened a new log
I20260927 22:14:33.072395 22820 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ff0f1e2a420845428fe8138c02394161"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58194
I20260927 22:14:33.081022 22845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } attempt: 4
I20260927 22:14:33.085593 22993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cb170e6f8ee478eae8cdb8ab294ec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0f1e2a420845428fe8138c02394161" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35327 } } peers { permanent_uuid: "9c1803db4add4de8849645932809c607" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } } peers { permanent_uuid: "2cb170e6f8ee478eae8cdb8ab294ec12" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } }
I20260927 22:14:33.086107 22993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cb170e6f8ee478eae8cdb8ab294ec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb170e6f8ee478eae8cdb8ab294ec12, State: Initialized, Role: FOLLOWER
I20260927 22:14:33.086712 22993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cb170e6f8ee478eae8cdb8ab294ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0f1e2a420845428fe8138c02394161" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35327 } } peers { permanent_uuid: "9c1803db4add4de8849645932809c607" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } } peers { permanent_uuid: "2cb170e6f8ee478eae8cdb8ab294ec12" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } }
I20260927 22:14:33.087775 23000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cb170e6f8ee478eae8cdb8ab294ec12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0f1e2a420845428fe8138c02394161" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35327 } } peers { permanent_uuid: "9c1803db4add4de8849645932809c607" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } } peers { permanent_uuid: "2cb170e6f8ee478eae8cdb8ab294ec12" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } } }
I20260927 22:14:33.088201 23000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cb170e6f8ee478eae8cdb8ab294ec12 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:14:33.088927 22993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cb170e6f8ee478eae8cdb8ab294ec12 [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 22:14:33 dist-test-slave-mp03 krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 22:14:33.098510 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:33.081732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40717 (local address 127.0.0.1:39472)
0927 22:14:33.082015 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:33.082063 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:33.082229 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:33.082632 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:33.082637 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:33.083120 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:33.083350 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:33.083364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.084911 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.084917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:33.086332 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:33.086346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.087410 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.087417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:33.087623 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:33.088293 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:33.088322 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:33.091094 (+ 2772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:33.095386 (+ 4292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:33.095391 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:33.095395 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:33.095909 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:33.096448 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:33.096456 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:33.096462 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:33.096656 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:33.097515 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:33.097520 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:33.097673 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 22:14:33.097826 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 22:14:33.098127 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
W20260927 22:14:33.099238 23011 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2cb170e6f8ee478eae8cdb8ab294ec12: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:14:33.099409 23011 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:14:33.099514 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:33.082104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39472 (local address 127.0.97.60:40717)
0927 22:14:33.082317 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:33.082324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:33.082344 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:14:33.082398 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:33.082405 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:33.082475 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:33.082571 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:33.083561 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.084752 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.086468 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.087266 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.087394 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:33.091286 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:33.091292 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:33.091297 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:33.091342 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:33.095209 (+ 3867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:33.096067 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:33.096071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:33.096074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:33.096264 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:33.096780 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:33.096784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:33.096787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:33.097253 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:33.097447 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:33.098828 (+ 1381us) server_negotiation.cc:299] Negotiation successful
0927 22:14:33.099059 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:14:33.105058 22845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161: Bootstrap starting.
I20260927 22:14:33.096365 22926 master_runner.cc:186] Error getting master registration for 127.0.97.62:35327: OK, instance_id { permanent_uuid: "ff0f1e2a420845428fe8138c02394161" instance_seqno: 1790547271936842 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:14:33.108453 22926 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:14:33.109120 22845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:33.110415 22845 log.cc:824] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:33.113123 22845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161: No bootstrap required, opened a new log
I20260927 22:14:33.118160 22845 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0f1e2a420845428fe8138c02394161" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35327 } } peers { permanent_uuid: "9c1803db4add4de8849645932809c607" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } } peers { permanent_uuid: "2cb170e6f8ee478eae8cdb8ab294ec12" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } }
I20260927 22:14:33.118510 22845 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff0f1e2a420845428fe8138c02394161, State: Initialized, Role: FOLLOWER
I20260927 22:14:33.119040 22845 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0f1e2a420845428fe8138c02394161" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35327 } } peers { permanent_uuid: "9c1803db4add4de8849645932809c607" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } } peers { permanent_uuid: "2cb170e6f8ee478eae8cdb8ab294ec12" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } }
I20260927 22:14:33.119989 23014 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0f1e2a420845428fe8138c02394161" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35327 } } peers { permanent_uuid: "9c1803db4add4de8849645932809c607" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } } peers { permanent_uuid: "2cb170e6f8ee478eae8cdb8ab294ec12" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } } }
I20260927 22:14:33.120380 23014 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:14:33.121286 22845 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:14:33.121379 22820 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 22:14:33.121618 22820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:33.125556 22820 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0f1e2a420845428fe8138c02394161" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35327 } } peers { permanent_uuid: "9c1803db4add4de8849645932809c607" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } } peers { permanent_uuid: "2cb170e6f8ee478eae8cdb8ab294ec12" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } }
I20260927 22:14:33.127194 22820 leader_election.cc:288] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161 [CANDIDATE]: Term 1 election: Requested vote from peers 9c1803db4add4de8849645932809c607 (127.0.97.61:39587), 2cb170e6f8ee478eae8cdb8ab294ec12 (127.0.97.60:40717)
I20260927 22:14:33.128412 22968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ff0f1e2a420845428fe8138c02394161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2cb170e6f8ee478eae8cdb8ab294ec12"
I20260927 22:14:33.128995 22968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cb170e6f8ee478eae8cdb8ab294ec12 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:33.133651 22968 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2cb170e6f8ee478eae8cdb8ab294ec12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff0f1e2a420845428fe8138c02394161 in term 1.
I20260927 22:14:33.134348 22795 leader_election.cc:302] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb170e6f8ee478eae8cdb8ab294ec12, ff0f1e2a420845428fe8138c02394161; no voters:
I20260927 22:14:33.134855 23014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:33.147946 23014 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161 [term 1 LEADER]: Becoming Leader. State: Replica: ff0f1e2a420845428fe8138c02394161, State: Running, Role: LEADER
I20260927 22:14:33.149150 23014 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0f1e2a420845428fe8138c02394161" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35327 } } peers { permanent_uuid: "9c1803db4add4de8849645932809c607" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } } peers { permanent_uuid: "2cb170e6f8ee478eae8cdb8ab294ec12" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } }
I20260927 22:14:33.168298 23021 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff0f1e2a420845428fe8138c02394161. Latest consensus state: current_term: 1 leader_uuid: "ff0f1e2a420845428fe8138c02394161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0f1e2a420845428fe8138c02394161" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35327 } } peers { permanent_uuid: "9c1803db4add4de8849645932809c607" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } } peers { permanent_uuid: "2cb170e6f8ee478eae8cdb8ab294ec12" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } } }
I20260927 22:14:33.168788 23021 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161 [sys.catalog]: This master's current role is: LEADER
W20260927 22:14:33.176009 22799 tablet.cc:2369] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:14:33.176927 22919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } attempt: 5
I20260927 22:14:33.180047 23031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 22:14:33 dist-test-slave-mp03 krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 22:14:33.194988 23031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:14:33.207005 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:33.177559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40717 (local address 127.0.0.1:39488)
0927 22:14:33.177841 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:33.177861 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:33.177982 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:33.180571 (+ 2589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:33.180576 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:33.181068 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:33.181267 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:33.181283 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.185377 (+ 4094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.185382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:33.186847 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:33.186863 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.189034 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.189060 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:33.189273 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:33.189950 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:33.189978 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:33.192883 (+ 2905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:33.202714 (+ 9831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:33.202720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:33.202724 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:33.203279 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:33.203794 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:33.203801 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:33.203807 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:33.203992 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:33.205921 (+ 1929us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:33.205927 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:33.206117 (+ 190us) client_negotiation.cc:769] Sending connection context
0927 22:14:33.206267 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 22:14:33.206579 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260927 22:14:33.208324 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:33.159909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58200 (local address 127.0.97.62:35327)
0927 22:14:33.160178 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:33.160188 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:33.172318 (+ 12130us) server_negotiation.cc:407] Connection header received
0927 22:14:33.172705 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:33.172715 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:33.172821 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:33.172915 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:33.179833 (+ 6918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.181101 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.183911 (+ 2810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.184588 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.184690 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:33.192339 (+ 7649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:33.192346 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:33.192351 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:33.192393 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:33.202343 (+ 9950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:33.204576 (+ 2233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:33.204585 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:33.204591 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:33.204786 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:33.205325 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:33.205333 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:33.205339 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:33.205790 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:33.206155 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:33.207413 (+ 1258us) server_negotiation.cc:299] Negotiation successful
0927 22:14:33.207776 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:14:33.215374 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:33.177718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39488 (local address 127.0.97.60:40717)
0927 22:14:33.180013 (+ 2295us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:33.180021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:33.180074 (+ 53us) server_negotiation.cc:407] Connection header received
0927 22:14:33.180165 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:33.180201 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:33.180324 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:33.181474 (+ 1150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:33.181879 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.184923 (+ 3044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.187290 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.188730 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.189731 (+ 1001us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:33.193337 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:33.193345 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:33.193350 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:33.193387 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:33.199295 (+ 5908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:33.203446 (+ 4151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:33.203456 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:33.203460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:33.203613 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:33.204145 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:33.204149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:33.204155 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:33.204501 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:33.214498 (+ 9997us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:33.214677 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 22:14:33.214917 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2099,"thread_start_us":1397,"threads_started":1}
I20260927 22:14:33.222612 22968 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2cb170e6f8ee478eae8cdb8ab294ec12 [term 1 FOLLOWER]: Refusing update from remote peer ff0f1e2a420845428fe8138c02394161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:14:33.223558 23014 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb170e6f8ee478eae8cdb8ab294ec12" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:14:33.226981 22947 tablet.cc:2369] T 00000000000000000000000000000000 P 2cb170e6f8ee478eae8cdb8ab294ec12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:14:33.223372 22919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c1803db4add4de8849645932809c607: Bootstrap starting.
I20260927 22:14:33.243855 22919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c1803db4add4de8849645932809c607: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:33.245262 22919 log.cc:824] T 00000000000000000000000000000000 P 9c1803db4add4de8849645932809c607: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:33.256330 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:33.177150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39476 (local address 127.0.97.60:40717)
0927 22:14:33.177380 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:33.177389 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:33.185069 (+ 7680us) server_negotiation.cc:407] Connection header received
0927 22:14:33.187428 (+ 2359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:33.187433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:33.187503 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:33.187576 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:33.193184 (+ 5608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.194370 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.199487 (+ 5117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.200218 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.200314 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:33.246216 (+ 45902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:33.246225 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:33.246231 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:33.246277 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:33.251339 (+ 5062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:33.253109 (+ 1770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:33.253116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:33.253120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:33.253267 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:33.254061 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:33.254067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:33.254070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:33.254441 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:33.254718 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:33.255684 (+ 966us) server_negotiation.cc:299] Negotiation successful
0927 22:14:33.255943 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 22:14:33.257230 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:33.171419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34506 (local address 127.0.97.61:39587)
0927 22:14:33.171678 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:33.171687 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:33.175870 (+ 4183us) server_negotiation.cc:407] Connection header received
0927 22:14:33.176742 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:33.176750 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:33.176845 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:33.176932 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:33.182283 (+ 5351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.183374 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.186797 (+ 3423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:33.187467 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:33.187587 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:33.248315 (+ 60728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:33.248322 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:33.248327 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:33.248369 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:33.252056 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:33.253569 (+ 1513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:33.253575 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:33.253579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:33.253732 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:33.254423 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:33.254428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:33.254432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:33.254842 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:33.255065 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:33.256004 (+ 939us) server_negotiation.cc:299] Negotiation successful
0927 22:14:33.256247 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 22:14:33.264453 22919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c1803db4add4de8849645932809c607: No bootstrap required, opened a new log
I20260927 22:14:33.226703 22778 master_runner.cc:186] Error getting master registration for 127.0.97.61:39587: OK, instance_id { permanent_uuid: "9c1803db4add4de8849645932809c607" instance_seqno: 1790547272448934 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:14:33.267364 22778 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:14:33.268723 23000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cb170e6f8ee478eae8cdb8ab294ec12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff0f1e2a420845428fe8138c02394161. Latest consensus state: current_term: 1 leader_uuid: "ff0f1e2a420845428fe8138c02394161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0f1e2a420845428fe8138c02394161" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35327 } } peers { permanent_uuid: "9c1803db4add4de8849645932809c607" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } } peers { permanent_uuid: "2cb170e6f8ee478eae8cdb8ab294ec12" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } } }
I20260927 22:14:33.269070 23000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cb170e6f8ee478eae8cdb8ab294ec12 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:14:33.269640 23038 mvcc.cc:205] Tried to move back new op lower bound from 7334081631097606144 to 7334081630830383104. Current Snapshot: MvccSnapshot[applied={T|T < 7334081631097606144}]
I20260927 22:14:33.281473 23031 catalog_manager.cc:1347] Generated new cluster ID: ced940f35f4749ac8e92a5dd8258bc20
I20260927 22:14:33.281596 23031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:14:33.286993 23021 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff0f1e2a420845428fe8138c02394161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0f1e2a420845428fe8138c02394161" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35327 } } peers { permanent_uuid: "9c1803db4add4de8849645932809c607" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } } peers { permanent_uuid: "2cb170e6f8ee478eae8cdb8ab294ec12" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } } }
I20260927 22:14:33.287304 23021 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161 [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:33.287730 23000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cb170e6f8ee478eae8cdb8ab294ec12 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff0f1e2a420845428fe8138c02394161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0f1e2a420845428fe8138c02394161" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35327 } } peers { permanent_uuid: "9c1803db4add4de8849645932809c607" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } } peers { permanent_uuid: "2cb170e6f8ee478eae8cdb8ab294ec12" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } } }
I20260927 22:14:33.287940 23000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cb170e6f8ee478eae8cdb8ab294ec12 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:14:33.291383 23039 mvcc.cc:205] Tried to move back new op lower bound from 7334081631097606144 to 7334081630830383104. Current Snapshot: MvccSnapshot[applied={T|T < 7334081631097606144}]
I20260927 22:14:33.291577 22919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c1803db4add4de8849645932809c607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0f1e2a420845428fe8138c02394161" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35327 } } peers { permanent_uuid: "9c1803db4add4de8849645932809c607" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } } peers { permanent_uuid: "2cb170e6f8ee478eae8cdb8ab294ec12" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } }
I20260927 22:14:33.292207 22919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c1803db4add4de8849645932809c607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c1803db4add4de8849645932809c607, State: Initialized, Role: FOLLOWER
I20260927 22:14:33.292878 22919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c1803db4add4de8849645932809c607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0f1e2a420845428fe8138c02394161" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35327 } } peers { permanent_uuid: "9c1803db4add4de8849645932809c607" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } } peers { permanent_uuid: "2cb170e6f8ee478eae8cdb8ab294ec12" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } }
I20260927 22:14:33.300201 23045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c1803db4add4de8849645932809c607 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0f1e2a420845428fe8138c02394161" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35327 } } peers { permanent_uuid: "9c1803db4add4de8849645932809c607" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } } peers { permanent_uuid: "2cb170e6f8ee478eae8cdb8ab294ec12" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } } }
I20260927 22:14:33.302738 23045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c1803db4add4de8849645932809c607 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:14:33.302940 22919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c1803db4add4de8849645932809c607 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:14:33.303287 22893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c1803db4add4de8849645932809c607 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:33.303408 22894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ff0f1e2a420845428fe8138c02394161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9c1803db4add4de8849645932809c607"
I20260927 22:14:33.308835 22893 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c1803db4add4de8849645932809c607 [term 1 FOLLOWER]: Refusing update from remote peer ff0f1e2a420845428fe8138c02394161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:14:33.314924 22894 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9c1803db4add4de8849645932809c607 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff0f1e2a420845428fe8138c02394161 in term 1.
I20260927 22:14:33.318964 23044 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1803db4add4de8849645932809c607" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:14:33.342615 23031 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:14:33.345901 23031 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:14:33.360309 23045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c1803db4add4de8849645932809c607 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff0f1e2a420845428fe8138c02394161. Latest consensus state: current_term: 1 leader_uuid: "ff0f1e2a420845428fe8138c02394161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0f1e2a420845428fe8138c02394161" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35327 } } peers { permanent_uuid: "9c1803db4add4de8849645932809c607" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } } peers { permanent_uuid: "2cb170e6f8ee478eae8cdb8ab294ec12" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } } }
I20260927 22:14:33.360366 23052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c1803db4add4de8849645932809c607 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff0f1e2a420845428fe8138c02394161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0f1e2a420845428fe8138c02394161" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35327 } } peers { permanent_uuid: "9c1803db4add4de8849645932809c607" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } } peers { permanent_uuid: "2cb170e6f8ee478eae8cdb8ab294ec12" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } } }
I20260927 22:14:33.360687 23045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c1803db4add4de8849645932809c607 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:14:33.360694 23052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c1803db4add4de8849645932809c607 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:14:33.362300 23014 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff0f1e2a420845428fe8138c02394161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0f1e2a420845428fe8138c02394161" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35327 } } peers { permanent_uuid: "9c1803db4add4de8849645932809c607" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39587 } } peers { permanent_uuid: "2cb170e6f8ee478eae8cdb8ab294ec12" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40717 } } }
I20260927 22:14:33.362617 23014 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161 [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:33.380679 23031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff0f1e2a420845428fe8138c02394161: Generated new TSK 0
I20260927 22:14:33.382025 23031 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:14:33.385696 23060 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260927 22:14:33.488656 22998 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:14:33.485960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58202 (local address 127.0.97.62:35327)
0927 22:14:33.486207 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:33.486214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:33.486790 (+ 576us) server_negotiation.cc:407] Connection header received
0927 22:14:33.487091 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:33.487097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:33.487185 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:33.487264 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:33.488403 (+ 1139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58202: BlockingRecv error: recv got EOF from 127.0.0.1:58202 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 22:14:33.491088 22923 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:14:33.486892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34508 (local address 127.0.97.61:39587)
0927 22:14:33.487089 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:33.487096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:33.489409 (+ 2313us) server_negotiation.cc:407] Connection header received
0927 22:14:33.489674 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:33.489680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:33.489759 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:33.489829 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:33.490874 (+ 1045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34508: BlockingRecv error: recv got EOF from 127.0.0.1:34508 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 22:14:33.493994 22996 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:14:33.492154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39498 (local address 127.0.97.60:40717)
0927 22:14:33.492316 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:33.492322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:33.492576 (+ 254us) server_negotiation.cc:407] Connection header received
0927 22:14:33.492722 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:33.492727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:33.492810 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:33.492916 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:33.493717 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39498: BlockingRecv error: recv got EOF from 127.0.0.1:39498 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 22:14:33.499905 22923 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:14:33.497919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34510 (local address 127.0.97.61:39587)
0927 22:14:33.498070 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:33.498076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:33.498547 (+ 471us) server_negotiation.cc:407] Connection header received
0927 22:14:33.498759 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:33.498764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:33.498891 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:33.498996 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:33.499678 (+ 682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34510: BlockingRecv error: recv got EOF from 127.0.0.1:34510 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 22:14:33.510663 22923 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:14:33.508604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34518 (local address 127.0.97.61:39587)
0927 22:14:33.508781 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:33.508787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:33.509251 (+ 464us) server_negotiation.cc:407] Connection header received
0927 22:14:33.509441 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:33.509447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:33.509542 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:33.509652 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:33.510422 (+ 770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34518: BlockingRecv error: recv got EOF from 127.0.0.1:34518 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 22:14:33.517959 22923 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:14:33.515822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34532 (local address 127.0.97.61:39587)
0927 22:14:33.515992 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:33.515997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:33.516552 (+ 555us) server_negotiation.cc:407] Connection header received
0927 22:14:33.516759 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:33.516764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:33.516859 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:33.516961 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:33.517728 (+ 767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34532: BlockingRecv error: recv got EOF from 127.0.0.1:34532 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 22:14:33.530515 22923 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:14:33.528232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34534 (local address 127.0.97.61:39587)
0927 22:14:33.528452 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:33.528458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:33.529015 (+ 557us) server_negotiation.cc:407] Connection header received
0927 22:14:33.529214 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:33.529220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:33.529313 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:33.529419 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:33.530276 (+ 857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34534: BlockingRecv error: recv got EOF from 127.0.0.1:34534 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
W20260927 22:14:33.544194 22923 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:14:33.541884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34550 (local address 127.0.97.61:39587)
0927 22:14:33.542072 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:33.542079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:33.542586 (+ 507us) server_negotiation.cc:407] Connection header received
0927 22:14:33.542895 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:33.542900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:33.543001 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:33.543124 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:33.543946 (+ 822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34550: BlockingRecv error: recv got EOF from 127.0.0.1:34550 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 22:14:33.566881 22923 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:14:33.564333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34554 (local address 127.0.97.61:39587)
0927 22:14:33.564527 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:33.564533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:33.565058 (+ 525us) server_negotiation.cc:407] Connection header received
0927 22:14:33.565422 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:33.565429 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:33.565553 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:33.565672 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:33.566583 (+ 911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34554: BlockingRecv error: recv got EOF from 127.0.0.1:34554 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 22:14:33.607584 22923 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:14:33.604898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34566 (local address 127.0.97.61:39587)
0927 22:14:33.605166 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:33.605175 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:33.605837 (+ 662us) server_negotiation.cc:407] Connection header received
0927 22:14:33.606057 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:33.606063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:33.606163 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:33.606287 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:33.607224 (+ 937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34566: BlockingRecv error: recv got EOF from 127.0.0.1:34566 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 22:14:33.678324 22923 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:14:33.676355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34578 (local address 127.0.97.61:39587)
0927 22:14:33.676589 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:33.676596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:33.676900 (+ 304us) server_negotiation.cc:407] Connection header received
0927 22:14:33.677086 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:33.677090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:33.677165 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:33.677254 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:33.678100 (+ 846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34578: BlockingRecv error: recv got EOF from 127.0.0.1:34578 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260927 22:14:33.811537 22923 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:14:33.809724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34592 (local address 127.0.97.61:39587)
0927 22:14:33.809910 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:33.809914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:33.810269 (+ 355us) server_negotiation.cc:407] Connection header received
0927 22:14:33.810455 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:33.810460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:33.810532 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:33.810707 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:33.811353 (+ 646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34592: BlockingRecv error: recv got EOF from 127.0.0.1:34592 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 22:14:34.074369 22923 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:14:34.072374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34600 (local address 127.0.97.61:39587)
0927 22:14:34.072583 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:34.072591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:34.072963 (+ 372us) server_negotiation.cc:407] Connection header received
0927 22:14:34.073212 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:34.073216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:34.073290 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:34.073409 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:34.074200 (+ 791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34600: BlockingRecv error: recv got EOF from 127.0.0.1:34600 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:14:34.101298 23011 catalog_manager.cc:1259] Loaded cluster ID: ced940f35f4749ac8e92a5dd8258bc20
I20260927 22:14:34.101395 23011 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2cb170e6f8ee478eae8cdb8ab294ec12: loading cluster ID for follower catalog manager: success
I20260927 22:14:34.104003 23011 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2cb170e6f8ee478eae8cdb8ab294ec12: acquiring CA information for follower catalog manager: success
I20260927 22:14:34.105659 23011 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2cb170e6f8ee478eae8cdb8ab294ec12: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:14:34.390110 23060 catalog_manager.cc:1259] Loaded cluster ID: ced940f35f4749ac8e92a5dd8258bc20
I20260927 22:14:34.390224 23060 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9c1803db4add4de8849645932809c607: loading cluster ID for follower catalog manager: success
I20260927 22:14:34.393081 23060 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9c1803db4add4de8849645932809c607: acquiring CA information for follower catalog manager: success
I20260927 22:14:34.395071 23060 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9c1803db4add4de8849645932809c607: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:14:34.600456 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:34.592496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34610 (local address 127.0.97.61:39587)
0927 22:14:34.593374 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:34.593382 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:34.593404 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:14:34.593632 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:34.593639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:34.593872 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:34.594130 (+ 258us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:14:34.594711 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:34.596016 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:34.597426 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:34.598182 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:34.598316 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:34.598963 (+ 647us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:14:34.599723 (+ 760us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:14:34.599874 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:34.599985 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 22:14:34.600078 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":538,"threads_started":1}
I20260927 22:14:34.613566 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:34.607064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39512 (local address 127.0.97.60:40717)
0927 22:14:34.607767 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:34.607772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:34.607788 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:14:34.607876 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:34.607879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:34.608029 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:34.608170 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:14:34.608617 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:34.609428 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:34.611352 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:34.611746 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:34.611839 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:34.612400 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:14:34.612921 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:14:34.613047 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:34.613175 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 22:14:34.613238 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":470,"threads_started":1}
I20260927 22:14:34.631685 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:34.622656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.0.97.61:39587)
0927 22:14:34.622836 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:34.622843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:34.623128 (+ 285us) server_negotiation.cc:407] Connection header received
0927 22:14:34.623405 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:34.623412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:34.623504 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:34.623582 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:14:34.624381 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:34.625896 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:34.628643 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:34.629593 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:34.629723 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:34.630550 (+ 827us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:14:34.630742 (+ 192us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:14:34.630819 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:34.631110 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 22:14:34.631234 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 22:14:34.638432 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:34.623514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39528 (local address 127.0.97.60:40717)
0927 22:14:34.623675 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:34.623682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:34.632856 (+ 9174us) server_negotiation.cc:407] Connection header received
0927 22:14:34.633187 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:34.633195 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:34.633340 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:34.633419 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:14:34.634123 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:34.635119 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:34.636749 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:34.637133 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:34.637271 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:34.637661 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:14:34.637766 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:14:34.637876 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:34.638037 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 22:14:34.638116 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:14:34.644052 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:34.621684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58212 (local address 127.0.97.62:35327)
0927 22:14:34.637749 (+ 16065us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:34.637757 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:34.637779 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:14:34.637919 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:34.637925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:34.638178 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:34.638389 (+ 211us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:14:34.638872 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:34.639954 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:34.641321 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:34.641932 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:34.642097 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:34.642881 (+ 784us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:14:34.643345 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:14:34.643470 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:34.643636 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 22:14:34.643728 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15915,"thread_start_us":16830,"threads_started":1}
I20260927 22:14:34.649475 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 22778
I20260927 22:14:34.650669 22840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:14:34.762411 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 22778
I20260927 22:14:34.783443 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 22852
I20260927 22:14:34.784600 22914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:14:34.892189 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 22852
I20260927 22:14:34.912528 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 22926
I20260927 22:14:34.913636 22988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:14:35.020387 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 22926
2026-09-27T22:14:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4897 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:14:35 dist-test-slave-mp03 krb5kdc[23145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:14:35 dist-test-slave-mp03 krb5kdc[23145](info): set up 2 sockets
Sep 27 22:14:35 dist-test-slave-mp03 krb5kdc[23145](info): commencing operation
krb5kdc: starting...
W20260927 22:14:36.147260 388 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/krb5kdc/test-user.keytab.
Sep 27 22:14:36 dist-test-slave-mp03 krb5kdc[23145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:14:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:14:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:14:36.408228 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46257
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46257,127.0.97.61:44909,127.0.97.60:35689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:36.741190 23158 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:14:36.741564 23158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:36.741691 23158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:36.741812 23158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:36.749249 23158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:36.749351 23158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:36.749445 23158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:36.749522 23158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:36.749611 23158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:36.759648 23158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46257,127.0.97.61:44909,127.0.97.60:35689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:36.761020 23158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:36.762360 23158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:36.773913 23166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:36.774075 23164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:36.774070 23163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:36.775103 23158 server_base.cc:1034] running on GCE node
I20260927 22:14:36.775838 23158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:36.777245 23158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:36.778465 23158 hybrid_clock.cc:648] HybridClock initialized: now 1790547276778392 us; error 42 us; skew 500 ppm
I20260927 22:14:36.778841 23158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:36 dist-test-slave-mp03 krb5kdc[23145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:36.782047 23158 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:14:36.783532 23158 webserver.cc:460] Webserver started at http://127.0.97.62:38065/ using document root <none> and password file <none>
I20260927 22:14:36.784211 23158 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:36.784296 23158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:36.784559 23158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:36.787554 23158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-0/data/instance:
uuid: "7683754b03af4cb19dc04fb3815dc76b"
format_stamp: "Formatted at 2026-09-27 22:14:36 on dist-test-slave-mp03"
server_key: "d46fea01b170033efb38f1eb2eabe06c"
server_key_iv: "86439975ab99356e1e47139667ee5462"
server_key_version: "encryptionkey@0"
I20260927 22:14:36.788318 23158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-0/wal/instance:
uuid: "7683754b03af4cb19dc04fb3815dc76b"
format_stamp: "Formatted at 2026-09-27 22:14:36 on dist-test-slave-mp03"
server_key: "d46fea01b170033efb38f1eb2eabe06c"
server_key_iv: "86439975ab99356e1e47139667ee5462"
server_key_version: "encryptionkey@0"
I20260927 22:14:36.793962 23158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:14:36.797534 23173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:36.798997 23158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:14:36.799197 23158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-0/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-0/wal
uuid: "7683754b03af4cb19dc04fb3815dc76b"
format_stamp: "Formatted at 2026-09-27 22:14:36 on dist-test-slave-mp03"
server_key: "d46fea01b170033efb38f1eb2eabe06c"
server_key_iv: "86439975ab99356e1e47139667ee5462"
server_key_version: "encryptionkey@0"
I20260927 22:14:36.799422 23158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:36.824610 23158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:36.829272 23158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:36.829566 23158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:36.849136 23158 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46257
I20260927 22:14:36.849176 23224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46257 every 8 connection(s)
I20260927 22:14:36.850606 23158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:14:36.853964 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 23158
I20260927 22:14:36.854166 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-0/wal/instance
I20260927 22:14:36.854748 388 external_mini_cluster.cc:1428] Setting key fe45c02b9b5a2914d112dbc10481ca46
I20260927 22:14:36.854977 23225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:36.865418 23225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:14:36 dist-test-slave-mp03 krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260927 22:14:36.889855 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:36.869004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45748 (local address 127.0.97.62:46257)
0927 22:14:36.870750 (+ 1746us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:36.870758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:36.870813 (+ 55us) server_negotiation.cc:407] Connection header received
0927 22:14:36.872402 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:36.872408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:36.872669 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:36.872907 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:36.874317 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:36.875626 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:36.877027 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:36.877593 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:36.877795 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:36.881800 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:36.881815 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:36.881835 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:36.881889 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:36.885336 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:36.886296 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:36.886302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:36.886305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:36.886453 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:36.886959 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:36.886963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:36.886965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:36.887532 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:36.887699 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:36.888441 (+ 742us) server_negotiation.cc:299] Negotiation successful
0927 22:14:36.888740 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1503,"thread_start_us":1296,"threads_started":1}
I20260927 22:14:36.889918 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:36.868983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46257 (local address 127.0.0.1:45748)
0927 22:14:36.869859 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:36.869907 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:36.872004 (+ 2097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:36.872921 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:36.872927 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:36.873716 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:36.874095 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:36.874114 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:36.875778 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:36.875783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:36.876878 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:36.876889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:36.877830 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:36.877837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:36.878008 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:36.878792 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:36.878815 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:36.881517 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:36.885498 (+ 3981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:36.885514 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:36.885530 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:36.886153 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:36.886571 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:36.886579 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:36.886586 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:36.886849 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:36.887683 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:36.887695 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:36.888011 (+ 316us) client_negotiation.cc:769] Sending connection context
0927 22:14:36.888307 (+ 296us) client_negotiation.cc:240] Negotiation successful
0927 22:14:36.888912 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":60,"thread_start_us":366,"threads_started":1}
I20260927 22:14:36.893123 23225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:14:36.894464 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:36.893912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44909 (local address 127.0.0.1:40546)
0927 22:14:36.894134 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:36.894279 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260927 22:14:36.894817 23175 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44909: connect: Connection refused (error 111)
I20260927 22:14:36.896576 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:36.896147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44909 (local address 127.0.0.1:40548)
0927 22:14:36.896316 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:36.896423 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260927 22:14:36.896812 23225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44909: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44909: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260927 22:14:36.926098 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44909
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46257,127.0.97.61:44909,127.0.97.60:35689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:14:36.946172 23225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } attempt: 1
I20260927 22:14:36.947438 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:36.946986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44909 (local address 127.0.0.1:40550)
0927 22:14:36.947190 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:36.947295 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 22:14:36.948570 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:36.948211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44909 (local address 127.0.0.1:40554)
0927 22:14:36.948360 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:36.948436 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 22:14:36.948762 23225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44909: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44909: connect: Connection refused (error 111)
I20260927 22:14:37.017040 23225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } attempt: 2
I20260927 22:14:37.018267 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.017760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44909 (local address 127.0.0.1:40568)
0927 22:14:37.018028 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.018133 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 22:14:37.019523 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.019022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44909 (local address 127.0.0.1:40582)
0927 22:14:37.019247 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.019355 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260927 22:14:37.019802 23225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44909: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44909: connect: Connection refused (error 111)
I20260927 22:14:37.111101 23225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } attempt: 3
I20260927 22:14:37.112572 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.111932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44909 (local address 127.0.0.1:40586)
0927 22:14:37.112214 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.112358 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 22:14:37.113729 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.113325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44909 (local address 127.0.0.1:40590)
0927 22:14:37.113475 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.113567 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 22:14:37.113946 23225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44909: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44909: connect: Connection refused (error 111)
I20260927 22:14:37.257221 23225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } attempt: 4
I20260927 22:14:37.258538 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.257907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44909 (local address 127.0.0.1:40592)
0927 22:14:37.258256 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.258368 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260927 22:14:37.259617 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.259214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44909 (local address 127.0.0.1:40596)
0927 22:14:37.259365 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.259464 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 22:14:37.259824 23225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44909: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44909: connect: Connection refused (error 111)
W20260927 22:14:37.273937 23232 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:14:37.274294 23232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:37.274366 23232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:37.274448 23232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:37.281761 23232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:37.281855 23232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:37.281912 23232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:37.281963 23232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:37.282027 23232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:37.292125 23232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46257,127.0.97.61:44909,127.0.97.60:35689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:37.293391 23232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:37.294782 23232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:37.307276 23237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:37.307232 23238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:37.307413 23240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:37.307978 23232 server_base.cc:1034] running on GCE node
I20260927 22:14:37.308800 23232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:37.310268 23232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:37.311553 23232 hybrid_clock.cc:648] HybridClock initialized: now 1790547277311472 us; error 50 us; skew 500 ppm
I20260927 22:14:37.311913 23232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:37 dist-test-slave-mp03 krb5kdc[23145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:37.315281 23232 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260927 22:14:37.316735 23232 webserver.cc:460] Webserver started at http://127.0.97.61:43349/ using document root <none> and password file <none>
I20260927 22:14:37.317445 23232 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:37.317571 23232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:37.317881 23232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:37.320851 23232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-1/data/instance:
uuid: "f6a0a601cbb64e239fb817c332faff68"
format_stamp: "Formatted at 2026-09-27 22:14:37 on dist-test-slave-mp03"
server_key: "b9c1ed181c31228e0579f395124d2a50"
server_key_iv: "cd2348400eb5ece1a66be3ef566d9007"
server_key_version: "encryptionkey@0"
I20260927 22:14:37.321651 23232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-1/wal/instance:
uuid: "f6a0a601cbb64e239fb817c332faff68"
format_stamp: "Formatted at 2026-09-27 22:14:37 on dist-test-slave-mp03"
server_key: "b9c1ed181c31228e0579f395124d2a50"
server_key_iv: "cd2348400eb5ece1a66be3ef566d9007"
server_key_version: "encryptionkey@0"
I20260927 22:14:37.326774 23232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:14:37.330199 23247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:37.331658 23232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:14:37.331857 23232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-1/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-1/wal
uuid: "f6a0a601cbb64e239fb817c332faff68"
format_stamp: "Formatted at 2026-09-27 22:14:37 on dist-test-slave-mp03"
server_key: "b9c1ed181c31228e0579f395124d2a50"
server_key_iv: "cd2348400eb5ece1a66be3ef566d9007"
server_key_version: "encryptionkey@0"
I20260927 22:14:37.332149 23232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:37.350894 23232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:37.355886 23232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:37.356199 23232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:37.376355 23232 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:44909
I20260927 22:14:37.376376 23298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:44909 every 8 connection(s)
I20260927 22:14:37.377800 23232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-1/data/info.pb
I20260927 22:14:37.381589 23299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:37.386175 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 23232
I20260927 22:14:37.386350 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-1/wal/instance
I20260927 22:14:37.386909 388 external_mini_cluster.cc:1428] Setting key 93ebc732361b08a42f53d9bf3867007a
I20260927 22:14:37.389308 23299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:14:37 dist-test-slave-mp03 krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:14:37.413532 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.392643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45760 (local address 127.0.97.62:46257)
0927 22:14:37.393150 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:37.393156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:37.394161 (+ 1005us) server_negotiation.cc:407] Connection header received
0927 22:14:37.397209 (+ 3048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:37.397216 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:37.397318 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:37.397435 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:37.399024 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.400213 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.401694 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.402419 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.402576 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:37.406165 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:37.406174 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:37.406181 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:37.406236 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:37.410046 (+ 3810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:37.410888 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:37.410896 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:37.410903 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:37.411083 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:37.411516 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:37.411523 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:37.411527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:37.411943 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:37.412130 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:37.412847 (+ 717us) server_negotiation.cc:299] Negotiation successful
0927 22:14:37.413111 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":238,"threads_started":1}
I20260927 22:14:37.414268 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.393175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46257 (local address 127.0.0.1:45760)
0927 22:14:37.393971 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.394020 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:37.396834 (+ 2814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:37.397623 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:37.397631 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:37.398197 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:37.398816 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:37.398835 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.400357 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.400361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:37.401555 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:37.401566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.402726 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.402731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:37.402918 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:37.403723 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:37.403758 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:37.405938 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:37.410233 (+ 4295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:37.410241 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:37.410254 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:37.410749 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:37.411216 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:37.411219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:37.411221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:37.411391 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:37.412081 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:37.412087 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:37.412447 (+ 360us) client_negotiation.cc:769] Sending connection context
0927 22:14:37.412684 (+ 237us) client_negotiation.cc:240] Negotiation successful
0927 22:14:37.413094 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":298,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260927 22:14:37.416095 23299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 27 22:14:37 dist-test-slave-mp03 krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:14:37.433203 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.416925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44909 (local address 127.0.0.1:40604)
0927 22:14:37.417208 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.417223 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:37.417328 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:37.418508 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:37.418513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:37.418948 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:37.419124 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:37.419132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.421195 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.421198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:37.422059 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:37.422071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.423295 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.423300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:37.423476 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:37.424094 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:37.424112 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:37.426520 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:37.430907 (+ 4387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:37.430911 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:37.430913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:37.431271 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:37.431682 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:37.431685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:37.431688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:37.431824 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:37.432521 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:37.432523 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:37.432629 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 22:14:37.432749 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 22:14:37.432936 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":36}
I20260927 22:14:37.433744 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.417278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40604 (local address 127.0.97.61:44909)
0927 22:14:37.417985 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:37.417994 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:37.418028 (+ 34us) server_negotiation.cc:407] Connection header received
0927 22:14:37.418088 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:37.418098 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:37.418339 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:37.418497 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:37.419300 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.421046 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.422197 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.423133 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.423268 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:37.426767 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:37.426783 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:37.426803 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:37.426844 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:37.430789 (+ 3945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:37.431401 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:37.431413 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:37.431419 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:37.431558 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:37.431936 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:37.431944 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:37.431949 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:37.432408 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:37.432611 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:37.432955 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 22:14:37.433241 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":534,"thread_start_us":380,"threads_started":1}
I20260927 22:14:37.436344 23299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:14:37.437548 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.437125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35689 (local address 127.0.0.1:34614)
0927 22:14:37.437292 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.437416 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 22:14:37.437815 23251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
I20260927 22:14:37.439334 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.439038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35689 (local address 127.0.0.1:34630)
0927 22:14:37.439165 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.439248 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 22:14:37.439544 23299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35689: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260927 22:14:37.453390 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3T1nfn/build/asan/bin/kudu
/tmp/dist-test-task3T1nfn/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35689
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46257,127.0.97.61:44909,127.0.97.60:35689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:14:37.488840 23299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } attempt: 1
I20260927 22:14:37.490063 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.489523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35689 (local address 127.0.0.1:34642)
0927 22:14:37.489754 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.489870 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 22:14:37.491178 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.490787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35689 (local address 127.0.0.1:34652)
0927 22:14:37.490953 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.491044 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 22:14:37.491511 23299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35689: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
I20260927 22:14:37.559077 23225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } attempt: 5
I20260927 22:14:37.559804 23299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } attempt: 2
I20260927 22:14:37.560847 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.560361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35689 (local address 127.0.0.1:34668)
0927 22:14:37.560507 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.560616 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 22:14:37.561915 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.561522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35689 (local address 127.0.0.1:34672)
0927 22:14:37.561675 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.561755 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 22:14:37.562119 23299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35689: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
Sep 27 22:14:37 dist-test-slave-mp03 krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:14:37.574633 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.559974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40606 (local address 127.0.97.61:44909)
0927 22:14:37.560347 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:37.560353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:37.560373 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:14:37.560568 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:37.560573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:37.560669 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:37.560764 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:37.561751 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.562800 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.564704 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.565133 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.565225 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:37.568661 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:37.568666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:37.568670 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:37.568701 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:37.571732 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:37.572375 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:37.572382 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:37.572386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:37.572532 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:37.573032 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:37.573039 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:37.573052 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:37.573446 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:37.573595 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:37.574063 (+ 468us) server_negotiation.cc:299] Negotiation successful
0927 22:14:37.574274 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242}
I20260927 22:14:37.574649 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.559901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44909 (local address 127.0.0.1:40606)
0927 22:14:37.560231 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.560254 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:37.560394 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:37.560832 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:37.560836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:37.561362 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:37.561620 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:37.561633 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.562969 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.562977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:37.564557 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:37.564573 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.565282 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.565288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:37.565518 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:37.566120 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:37.566142 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:37.568464 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:37.571866 (+ 3402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:37.571872 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:37.571876 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:37.572249 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:37.572680 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:37.572726 (+ 46us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:37.572732 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:37.572900 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:37.573591 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:37.573595 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:37.573783 (+ 188us) client_negotiation.cc:769] Sending connection context
0927 22:14:37.573944 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 22:14:37.574268 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260927 22:14:37.576112 23225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:14:37.577268 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.576799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35689 (local address 127.0.0.1:34688)
0927 22:14:37.576991 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.577125 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 22:14:37.578161 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.577828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35689 (local address 127.0.0.1:34700)
0927 22:14:37.577982 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.578054 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 22:14:37.578364 23225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35689: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
I20260927 22:14:37.629650 23225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } attempt: 1
I20260927 22:14:37.630880 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.630356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35689 (local address 127.0.0.1:34708)
0927 22:14:37.630563 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.630714 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 22:14:37.632053 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.631737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35689 (local address 127.0.0.1:34720)
0927 22:14:37.631881 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.631951 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 22:14:37.632380 23225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35689: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
I20260927 22:14:37.653362 23299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } attempt: 3
I20260927 22:14:37.654583 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.654077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35689 (local address 127.0.0.1:34726)
0927 22:14:37.654315 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.654409 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 22:14:37.655773 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.655342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35689 (local address 127.0.0.1:34728)
0927 22:14:37.655575 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.655666 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260927 22:14:37.656148 23299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35689: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
I20260927 22:14:37.700611 23225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } attempt: 2
I20260927 22:14:37.701946 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.701339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35689 (local address 127.0.0.1:34736)
0927 22:14:37.701641 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.701769 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 22:14:37.702927 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.702565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35689 (local address 127.0.0.1:34742)
0927 22:14:37.702748 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.702814 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260927 22:14:37.703120 23225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35689: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
W20260927 22:14:37.788395 23307 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:14:37.788785 23307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:37.788916 23307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:37.789108 23307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:37.796411 23307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:37.796536 23307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:37.796645 23307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:37.796734 23307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:37.796823 23307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:37.799405 23299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } attempt: 4
I20260927 22:14:37.800532 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.800055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35689 (local address 127.0.0.1:34744)
0927 22:14:37.800285 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.800379 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 22:14:37.801461 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.801138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35689 (local address 127.0.0.1:34754)
0927 22:14:37.801289 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.801368 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 22:14:37.801662 23299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35689: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
I20260927 22:14:37.807447 23307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46257,127.0.97.61:44909,127.0.97.60:35689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:37.808838 23307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:37.809324 23225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } attempt: 3
I20260927 22:14:37.810235 23307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 22:14:37.810277 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.809853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35689 (local address 127.0.0.1:34756)
0927 22:14:37.810032 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.810129 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 22:14:37.811328 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.810926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35689 (local address 127.0.0.1:34764)
0927 22:14:37.811163 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.811236 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260927 22:14:37.811512 23225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35689: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35689: connect: Connection refused (error 111)
W20260927 22:14:37.822549 23315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:37.822675 23312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:37.822521 23313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:37.823318 23307 server_base.cc:1034] running on GCE node
I20260927 22:14:37.824034 23307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:37.825344 23307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:37.826566 23307 hybrid_clock.cc:648] HybridClock initialized: now 1790547277826508 us; error 41 us; skew 500 ppm
I20260927 22:14:37.826964 23307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:37 dist-test-slave-mp03 krb5kdc[23145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:37.830041 23307 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260927 22:14:37.831456 23307 webserver.cc:460] Webserver started at http://127.0.97.60:46825/ using document root <none> and password file <none>
I20260927 22:14:37.832123 23307 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:37.832234 23307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:37.832573 23307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:37.835639 23307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-2/data/instance:
uuid: "1ca15165cd77409daae3d8d3a289bad3"
format_stamp: "Formatted at 2026-09-27 22:14:37 on dist-test-slave-mp03"
server_key: "c74d42ea9bec49a6bfa3a2c1b04ebf18"
server_key_iv: "c76c6b144578b70fc5133d80a7fb6c02"
server_key_version: "encryptionkey@0"
I20260927 22:14:37.836519 23307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-2/wal/instance:
uuid: "1ca15165cd77409daae3d8d3a289bad3"
format_stamp: "Formatted at 2026-09-27 22:14:37 on dist-test-slave-mp03"
server_key: "c74d42ea9bec49a6bfa3a2c1b04ebf18"
server_key_iv: "c76c6b144578b70fc5133d80a7fb6c02"
server_key_version: "encryptionkey@0"
I20260927 22:14:37.841544 23307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 22:14:37.845327 23322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:37.846726 23307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:14:37.846897 23307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-2/data,/tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-2/wal
uuid: "1ca15165cd77409daae3d8d3a289bad3"
format_stamp: "Formatted at 2026-09-27 22:14:37 on dist-test-slave-mp03"
server_key: "c74d42ea9bec49a6bfa3a2c1b04ebf18"
server_key_iv: "c76c6b144578b70fc5133d80a7fb6c02"
server_key_version: "encryptionkey@0"
I20260927 22:14:37.847112 23307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:37.874045 23307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:37.878762 23307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:37.879053 23307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:37.897607 23307 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:35689
I20260927 22:14:37.897629 23373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:35689 every 8 connection(s)
I20260927 22:14:37.899063 23307 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-2/data/info.pb
I20260927 22:14:37.902805 23374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:37.907444 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3T1nfn/build/asan/bin/kudu as pid 23307
I20260927 22:14:37.907625 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3T1nfn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546912983837-388-0/minicluster-data/master-2/wal/instance
I20260927 22:14:37.908162 388 external_mini_cluster.cc:1428] Setting key ed6768c0b1c6638c958988eb9a649532
I20260927 22:14:37.910254 23374 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:14:37 dist-test-slave-mp03 krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 27 22:14:37 dist-test-slave-mp03 krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 22:14:37.929479 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.912986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45768 (local address 127.0.97.62:46257)
0927 22:14:37.913230 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:37.913240 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:37.913921 (+ 681us) server_negotiation.cc:407] Connection header received
0927 22:14:37.914201 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:37.914210 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:37.914337 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:37.914457 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:37.916041 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.916712 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.918317 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.918830 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.918923 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:37.922453 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:37.922460 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:37.922466 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:37.922504 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:37.925847 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:37.926863 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:37.926867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:37.926870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:37.926992 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:37.927603 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:37.927611 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:37.927615 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:37.927961 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:37.928108 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:37.928903 (+ 795us) server_negotiation.cc:299] Negotiation successful
0927 22:14:37.929131 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:14:37.930473 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.909990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34774 (local address 127.0.97.60:35689)
0927 22:14:37.910863 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:37.910875 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:37.910920 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:14:37.912644 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:37.912667 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:37.913227 (+ 560us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:37.913701 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:37.915353 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.916982 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.918403 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.919017 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.919196 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:37.922968 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:37.922977 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:37.922994 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:37.923041 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:37.926374 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:37.927381 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:37.927387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:37.927389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:37.927518 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:37.927928 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:37.927932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:37.927936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:37.928407 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:37.928628 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:37.929011 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 22:14:37.929633 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":443,"server-negotiator.queue_time_us":659,"thread_start_us":387,"threads_started":1}
I20260927 22:14:37.930513 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.912968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46257 (local address 127.0.0.1:45768)
0927 22:14:37.913763 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.913823 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:37.914038 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:37.914514 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:37.914519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:37.915212 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:37.915868 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:37.915887 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.916860 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.916866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:37.918166 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:37.918177 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.919048 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.919056 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:37.919260 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:37.920126 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:37.920150 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:37.922270 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:37.926015 (+ 3745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:37.926025 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:37.926042 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:37.926736 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:37.927247 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:37.927253 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:37.927256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:37.927492 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:37.928088 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:37.928095 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:37.928448 (+ 353us) client_negotiation.cc:769] Sending connection context
0927 22:14:37.928783 (+ 335us) client_negotiation.cc:240] Negotiation successful
0927 22:14:37.929318 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":125,"spinlock_wait_cycles":14592,"thread_start_us":386,"threads_started":1}
I20260927 22:14:37.932181 23374 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:14:37 dist-test-slave-mp03 krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 27 22:14:37 dist-test-slave-mp03 krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:14:37.946887 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.932942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40614 (local address 127.0.97.61:44909)
0927 22:14:37.933123 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:37.933130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:37.933212 (+ 82us) server_negotiation.cc:407] Connection header received
0927 22:14:37.933337 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:37.933343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:37.933424 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:37.933546 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:37.934375 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.935158 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.936685 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.937140 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.937256 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:37.940766 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:37.940784 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:37.940787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:37.940814 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:37.944168 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:37.944943 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:37.944949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:37.944953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:37.945120 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:37.945563 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:37.945568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:37.945571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:37.945916 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:37.946066 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:37.946385 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 22:14:37.946564 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 22:14:37.946996 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.932962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44909 (local address 127.0.0.1:40614)
0927 22:14:37.933150 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.933166 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:37.933251 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:37.933577 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:37.933581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:37.934014 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:37.934187 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:37.934219 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.935305 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.935311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:37.936544 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:37.936558 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.937291 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.937297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:37.937490 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:37.938130 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:37.938161 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:37.940584 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:37.944297 (+ 3713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:37.944301 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:37.944304 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:37.944818 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:37.945237 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:37.945240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:37.945242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:37.945426 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:37.946037 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:37.946040 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:37.946156 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 22:14:37.946261 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 22:14:37.946539 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 22:14:37.948052 23374 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:14:37.949474 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.933718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45782 (local address 127.0.97.62:46257)
0927 22:14:37.933874 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:37.933878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:37.933892 (+ 14us) server_negotiation.cc:407] Connection header received
0927 22:14:37.934081 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:37.934085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:37.934149 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:37.934227 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:37.935454 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.936187 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.937651 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.938144 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.938270 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:37.942785 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:37.942793 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:37.942798 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:37.942837 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:37.946233 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:37.947055 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:37.947060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:37.947063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:37.947364 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:37.947884 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:37.947889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:37.947898 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:37.948165 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:37.948431 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:37.948899 (+ 468us) server_negotiation.cc:299] Negotiation successful
0927 22:14:37.949119 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 22:14:37 dist-test-slave-mp03 krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 22:14:37 dist-test-slave-mp03 krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:14:37.961593 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.948844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35689 (local address 127.0.0.1:34776)
0927 22:14:37.949035 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.949056 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:37.949147 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:37.949452 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:37.949456 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:37.949849 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:37.949987 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:37.949995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.951213 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.951219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:37.952423 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:37.952434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.953143 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.953149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:37.953357 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:37.953916 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:37.953931 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:37.956108 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:37.959007 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:37.959012 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:37.959017 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:37.959526 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:37.959867 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:37.959874 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:37.959879 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:37.960002 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:37.960583 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:37.960588 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:37.960734 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 22:14:37.960898 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 22:14:37.961188 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 22:14:37.961627 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.949046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34776 (local address 127.0.97.60:35689)
0927 22:14:37.949185 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:37.949191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:37.949207 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:14:37.949252 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:37.949259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:37.949325 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:37.949419 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:37.950159 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.951052 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.952554 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.953003 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.953184 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:37.956281 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:37.956289 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:37.956297 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:37.956336 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:37.958882 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:37.959627 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:37.959630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:37.959632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:37.959766 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:37.960105 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:37.960109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:37.960118 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:37.960465 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:37.960619 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:37.960979 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 22:14:37.961329 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":143,"server-negotiator.queue_time_us":49}
I20260927 22:14:37.964962 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.951150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40616 (local address 127.0.97.61:44909)
0927 22:14:37.951325 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:37.951331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:37.951378 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:14:37.951492 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:37.951497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:37.951576 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:37.951674 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:37.952902 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.953795 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.955014 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.955424 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.955561 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:37.959766 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:37.959771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:37.959774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:37.959802 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:37.962523 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:37.963139 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:37.963148 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:37.963151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:37.963286 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:37.963671 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:37.963678 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:37.963681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:37.963922 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:37.964249 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:37.964439 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 22:14:37.964664 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:14:37.966717 23374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ca15165cd77409daae3d8d3a289bad3: Bootstrap starting.
I20260927 22:14:37.969807 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:14:37.970173 23374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ca15165cd77409daae3d8d3a289bad3: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:37.971315 23374 log.cc:824] T 00000000000000000000000000000000 P 1ca15165cd77409daae3d8d3a289bad3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:37.974475 23374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ca15165cd77409daae3d8d3a289bad3: No bootstrap required, opened a new log
I20260927 22:14:37.972198 23200 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7683754b03af4cb19dc04fb3815dc76b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45782
I20260927 22:14:37.981812 23374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ca15165cd77409daae3d8d3a289bad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7683754b03af4cb19dc04fb3815dc76b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } } peers { permanent_uuid: "f6a0a601cbb64e239fb817c332faff68" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } } peers { permanent_uuid: "1ca15165cd77409daae3d8d3a289bad3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } }
I20260927 22:14:37.982136 23374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ca15165cd77409daae3d8d3a289bad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ca15165cd77409daae3d8d3a289bad3, State: Initialized, Role: FOLLOWER
I20260927 22:14:37.982678 23374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ca15165cd77409daae3d8d3a289bad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7683754b03af4cb19dc04fb3815dc76b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } } peers { permanent_uuid: "f6a0a601cbb64e239fb817c332faff68" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } } peers { permanent_uuid: "1ca15165cd77409daae3d8d3a289bad3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } }
I20260927 22:14:37.983645 23380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ca15165cd77409daae3d8d3a289bad3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7683754b03af4cb19dc04fb3815dc76b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } } peers { permanent_uuid: "f6a0a601cbb64e239fb817c332faff68" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } } peers { permanent_uuid: "1ca15165cd77409daae3d8d3a289bad3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } } }
I20260927 22:14:37.984030 23380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ca15165cd77409daae3d8d3a289bad3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:14:37.984865 23374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ca15165cd77409daae3d8d3a289bad3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:14:37.988718 23225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } attempt: 4
W20260927 22:14:37.995391 23391 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ca15165cd77409daae3d8d3a289bad3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:14:37.995599 23391 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 27 22:14:37 dist-test-slave-mp03 krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 22:14:37.993111 23307 master_runner.cc:186] Error getting master registration for 127.0.97.62:46257: OK, instance_id { permanent_uuid: "7683754b03af4cb19dc04fb3815dc76b" instance_seqno: 1790547276843700 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:14:38.005015 23307 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:14:38.006518 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.989418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35689 (local address 127.0.0.1:34782)
0927 22:14:37.989601 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:37.989623 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:37.989757 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:37.990231 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:37.990236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:37.990753 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:37.990954 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:37.990967 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.992419 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.992424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:37.993942 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:37.993958 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.995405 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.995412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:37.995611 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:37.996308 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:37.996333 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:37.999197 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:38.003511 (+ 4314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:38.003517 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:38.003521 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:38.004055 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:38.004549 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:38.004567 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:38.004571 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:38.004736 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:38.005461 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:38.005467 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:38.005644 (+ 177us) client_negotiation.cc:769] Sending connection context
0927 22:14:38.005809 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 22:14:38.006134 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 22:14:38.006522 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:37.989597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34782 (local address 127.0.97.60:35689)
0927 22:14:37.989831 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:37.989840 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:37.989868 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:14:37.989942 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:37.989950 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:37.990047 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:37.990169 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:37.991104 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.992249 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.994162 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:37.994998 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:37.995131 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:37.999418 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:37.999430 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:37.999436 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:37.999483 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:38.003354 (+ 3871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:38.004230 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:38.004236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:38.004240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:38.004411 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:38.004869 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:38.004874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:38.004880 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:38.005308 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:38.005486 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:38.005930 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 22:14:38.006162 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:14:38.012866 23225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b: Bootstrap starting.
I20260927 22:14:38.017176 23225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:38.018678 23225 log.cc:824] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:38.022076 23225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b: No bootstrap required, opened a new log
I20260927 22:14:38.029853 23225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7683754b03af4cb19dc04fb3815dc76b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } } peers { permanent_uuid: "f6a0a601cbb64e239fb817c332faff68" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } } peers { permanent_uuid: "1ca15165cd77409daae3d8d3a289bad3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } }
I20260927 22:14:38.030304 23225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7683754b03af4cb19dc04fb3815dc76b, State: Initialized, Role: FOLLOWER
I20260927 22:14:38.030845 23225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7683754b03af4cb19dc04fb3815dc76b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } } peers { permanent_uuid: "f6a0a601cbb64e239fb817c332faff68" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } } peers { permanent_uuid: "1ca15165cd77409daae3d8d3a289bad3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } }
I20260927 22:14:38.031865 23394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7683754b03af4cb19dc04fb3815dc76b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } } peers { permanent_uuid: "f6a0a601cbb64e239fb817c332faff68" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } } peers { permanent_uuid: "1ca15165cd77409daae3d8d3a289bad3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } } }
I20260927 22:14:38.032269 23394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:14:38.033138 23225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:14:38.033198 23200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 22:14:38.033416 23200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:38.036682 23200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7683754b03af4cb19dc04fb3815dc76b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } } peers { permanent_uuid: "f6a0a601cbb64e239fb817c332faff68" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } } peers { permanent_uuid: "1ca15165cd77409daae3d8d3a289bad3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } }
I20260927 22:14:38.039492 23349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7683754b03af4cb19dc04fb3815dc76b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1ca15165cd77409daae3d8d3a289bad3"
I20260927 22:14:38.040068 23349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ca15165cd77409daae3d8d3a289bad3 [term 0 FOLLOWER]: Advancing to term 1
W20260927 22:14:38.044895 23405 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:14:38.045024 23405 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:14:38.046362 23349 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1ca15165cd77409daae3d8d3a289bad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7683754b03af4cb19dc04fb3815dc76b in term 1.
I20260927 22:14:38.047031 23177 leader_election.cc:302] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ca15165cd77409daae3d8d3a289bad3, 7683754b03af4cb19dc04fb3815dc76b; no voters:
I20260927 22:14:38.047614 23394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:38.053396 23200 leader_election.cc:288] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b [CANDIDATE]: Term 1 election: Requested vote from peers f6a0a601cbb64e239fb817c332faff68 (127.0.97.61:44909), 1ca15165cd77409daae3d8d3a289bad3 (127.0.97.60:35689)
I20260927 22:14:38.061650 23394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b [term 1 LEADER]: Becoming Leader. State: Replica: 7683754b03af4cb19dc04fb3815dc76b, State: Running, Role: LEADER
I20260927 22:14:38.062346 23394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7683754b03af4cb19dc04fb3815dc76b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } } peers { permanent_uuid: "f6a0a601cbb64e239fb817c332faff68" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } } peers { permanent_uuid: "1ca15165cd77409daae3d8d3a289bad3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } }
I20260927 22:14:38.047185 23158 master_runner.cc:186] Error getting master registration for 127.0.97.61:44909: OK, instance_id { permanent_uuid: "f6a0a601cbb64e239fb817c332faff68" instance_seqno: 1790547277370556 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:14:38.069715 23158 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:14:38.076116 23407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7683754b03af4cb19dc04fb3815dc76b. Latest consensus state: current_term: 1 leader_uuid: "7683754b03af4cb19dc04fb3815dc76b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7683754b03af4cb19dc04fb3815dc76b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } } peers { permanent_uuid: "f6a0a601cbb64e239fb817c332faff68" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } } peers { permanent_uuid: "1ca15165cd77409daae3d8d3a289bad3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } } }
I20260927 22:14:38.076692 23407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b [sys.catalog]: This master's current role is: LEADER
W20260927 22:14:38.082559 23179 tablet.cc:2369] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:14:38.088037 23411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:14:38.090338 23411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:14:38.100860 23299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } attempt: 5
I20260927 22:14:38.109570 23349 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ca15165cd77409daae3d8d3a289bad3 [term 1 FOLLOWER]: Refusing update from remote peer 7683754b03af4cb19dc04fb3815dc76b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:14:38.111976 23407 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ca15165cd77409daae3d8d3a289bad3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 27 22:14:38 dist-test-slave-mp03 krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260927 22:14:38.131170 23328 tablet.cc:2369] T 00000000000000000000000000000000 P 1ca15165cd77409daae3d8d3a289bad3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:14:38.146373 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:38.101586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35689 (local address 127.0.0.1:34798)
0927 22:14:38.101969 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:38.101989 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:38.102111 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:38.103438 (+ 1327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:38.103443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:38.104001 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:38.104364 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:38.104380 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:38.105936 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:38.105942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:38.107448 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:38.107464 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:38.117416 (+ 9952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:38.117424 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:38.117662 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:38.118994 (+ 1332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:38.119020 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:38.123844 (+ 4824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:38.130824 (+ 6980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:38.130831 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:38.130835 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:38.131941 (+ 1106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:38.133484 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:38.133490 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:38.133496 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:38.134444 (+ 948us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:38.136674 (+ 2230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:38.136681 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:38.137937 (+ 1256us) client_negotiation.cc:769] Sending connection context
0927 22:14:38.140783 (+ 2846us) client_negotiation.cc:240] Negotiation successful
0927 22:14:38.143051 (+ 2268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258}
I20260927 22:14:38.151010 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:38.102348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34798 (local address 127.0.97.60:35689)
0927 22:14:38.102907 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:38.102914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:38.102934 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:14:38.103000 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:38.103005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:38.103100 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:38.103210 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:38.104565 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:38.105718 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:38.107676 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:38.108373 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:38.108520 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:38.124258 (+ 15738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:38.124267 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:38.124272 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:38.124316 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:38.130424 (+ 6108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:38.132207 (+ 1783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:38.132213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:38.132217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:38.132547 (+ 330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:38.134937 (+ 2390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:38.134943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:38.134947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:38.135695 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:38.136589 (+ 894us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:38.148733 (+ 12144us) server_negotiation.cc:299] Negotiation successful
0927 22:14:38.149254 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436}
I20260927 22:14:38.158026 23299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6a0a601cbb64e239fb817c332faff68: Bootstrap starting.
I20260927 22:14:38.164126 23299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6a0a601cbb64e239fb817c332faff68: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:38.177922 23299 log.cc:824] T 00000000000000000000000000000000 P f6a0a601cbb64e239fb817c332faff68: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:38.180280 23380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ca15165cd77409daae3d8d3a289bad3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7683754b03af4cb19dc04fb3815dc76b. Latest consensus state: current_term: 1 leader_uuid: "7683754b03af4cb19dc04fb3815dc76b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7683754b03af4cb19dc04fb3815dc76b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } } peers { permanent_uuid: "f6a0a601cbb64e239fb817c332faff68" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } } peers { permanent_uuid: "1ca15165cd77409daae3d8d3a289bad3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } } }
I20260927 22:14:38.180630 23380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ca15165cd77409daae3d8d3a289bad3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:14:38.185083 23380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ca15165cd77409daae3d8d3a289bad3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7683754b03af4cb19dc04fb3815dc76b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7683754b03af4cb19dc04fb3815dc76b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } } peers { permanent_uuid: "f6a0a601cbb64e239fb817c332faff68" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } } peers { permanent_uuid: "1ca15165cd77409daae3d8d3a289bad3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } } }
I20260927 22:14:38.185370 23380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ca15165cd77409daae3d8d3a289bad3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:14:38.204952 23394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7683754b03af4cb19dc04fb3815dc76b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7683754b03af4cb19dc04fb3815dc76b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } } peers { permanent_uuid: "f6a0a601cbb64e239fb817c332faff68" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } } peers { permanent_uuid: "1ca15165cd77409daae3d8d3a289bad3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } } }
I20260927 22:14:38.205303 23394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:38.218917 23299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6a0a601cbb64e239fb817c332faff68: No bootstrap required, opened a new log
I20260927 22:14:38.240607 23411 catalog_manager.cc:1347] Generated new cluster ID: 698c79eeb1474f81ba457b954bfe3c89
I20260927 22:14:38.240744 23411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:14:38.248224 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:38.110726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34804 (local address 127.0.97.60:35689)
0927 22:14:38.165785 (+ 55059us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:38.165796 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:38.197253 (+ 31457us) server_negotiation.cc:407] Connection header received
0927 22:14:38.197399 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:38.197405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:38.197658 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:38.197863 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:38.214352 (+ 16489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:38.215539 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:38.229204 (+ 13665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:38.229916 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:38.230035 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:38.238338 (+ 8303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:38.238346 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:38.238351 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:38.238398 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:38.243225 (+ 4827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:38.244100 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:38.244106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:38.244111 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:38.244273 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:38.244727 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:38.244732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:38.244736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:38.245089 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:38.245361 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:38.247303 (+ 1942us) server_negotiation.cc:299] Negotiation successful
0927 22:14:38.247616 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40694,"thread_start_us":124565,"threads_started":1}
I20260927 22:14:38.261788 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:38.096460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45786 (local address 127.0.97.62:46257)
0927 22:14:38.096645 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:38.096652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:38.110202 (+ 13550us) server_negotiation.cc:407] Connection header received
0927 22:14:38.110268 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:38.110276 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:38.110381 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:38.110482 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:38.174028 (+ 63546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:38.175301 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:38.236448 (+ 61147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:38.237239 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:38.237377 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:38.253922 (+ 16545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:38.253930 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:38.253934 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:38.253980 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:38.257922 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:38.258873 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:38.258882 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:38.258889 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:38.259076 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:38.259573 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:38.259581 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:38.259586 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:38.259994 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:38.260274 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:38.260934 (+ 660us) server_negotiation.cc:299] Negotiation successful
0927 22:14:38.261199 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:14:38.274796 23419 mvcc.cc:205] Tried to move back new op lower bound from 7334081651121627136 to 7334081650958729216. Current Snapshot: MvccSnapshot[applied={T|T < 7334081651121627136}]
I20260927 22:14:38.276067 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:38.103035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40624 (local address 127.0.97.61:44909)
0927 22:14:38.103197 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:38.103203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:38.196449 (+ 93246us) server_negotiation.cc:407] Connection header received
0927 22:14:38.196800 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:38.196808 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:38.196906 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:38.197008 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:38.234571 (+ 37563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:38.235919 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:38.238284 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:38.239072 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:38.239190 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:38.268023 (+ 28833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:38.268031 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:38.268036 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:38.268088 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:38.272081 (+ 3993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:38.272974 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:38.272981 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:38.272985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:38.273171 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:38.273607 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:38.273617 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:38.273621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:38.273997 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:38.274287 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:38.274831 (+ 544us) server_negotiation.cc:299] Negotiation successful
0927 22:14:38.275281 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:14:38.280215 23299 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6a0a601cbb64e239fb817c332faff68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7683754b03af4cb19dc04fb3815dc76b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } } peers { permanent_uuid: "f6a0a601cbb64e239fb817c332faff68" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } } peers { permanent_uuid: "1ca15165cd77409daae3d8d3a289bad3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } }
I20260927 22:14:38.283670 23299 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6a0a601cbb64e239fb817c332faff68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a0a601cbb64e239fb817c332faff68, State: Initialized, Role: FOLLOWER
I20260927 22:14:38.284296 23299 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6a0a601cbb64e239fb817c332faff68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7683754b03af4cb19dc04fb3815dc76b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } } peers { permanent_uuid: "f6a0a601cbb64e239fb817c332faff68" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } } peers { permanent_uuid: "1ca15165cd77409daae3d8d3a289bad3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } }
I20260927 22:14:38.286087 23424 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6a0a601cbb64e239fb817c332faff68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7683754b03af4cb19dc04fb3815dc76b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } } peers { permanent_uuid: "f6a0a601cbb64e239fb817c332faff68" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } } peers { permanent_uuid: "1ca15165cd77409daae3d8d3a289bad3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } } }
I20260927 22:14:38.287173 23424 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6a0a601cbb64e239fb817c332faff68 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:14:38.289321 23299 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6a0a601cbb64e239fb817c332faff68 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:14:38.295591 23411 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:14:38.298789 23411 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:14:38.311180 23273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6a0a601cbb64e239fb817c332faff68 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:38.317443 23411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b: Generated new TSK 0
I20260927 22:14:38.317932 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7683754b03af4cb19dc04fb3815dc76b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f6a0a601cbb64e239fb817c332faff68"
I20260927 22:14:38.318640 23411 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:14:38.320138 23273 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6a0a601cbb64e239fb817c332faff68 [term 1 FOLLOWER]: Refusing update from remote peer 7683754b03af4cb19dc04fb3815dc76b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:14:38.321117 23407 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a0a601cbb64e239fb817c332faff68" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:14:38.323841 23274 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f6a0a601cbb64e239fb817c332faff68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7683754b03af4cb19dc04fb3815dc76b in term 1.
W20260927 22:14:38.360565 23253 tablet.cc:2369] T 00000000000000000000000000000000 P f6a0a601cbb64e239fb817c332faff68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:14:38.398321 23424 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6a0a601cbb64e239fb817c332faff68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7683754b03af4cb19dc04fb3815dc76b. Latest consensus state: current_term: 1 leader_uuid: "7683754b03af4cb19dc04fb3815dc76b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7683754b03af4cb19dc04fb3815dc76b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } } peers { permanent_uuid: "f6a0a601cbb64e239fb817c332faff68" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } } peers { permanent_uuid: "1ca15165cd77409daae3d8d3a289bad3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } } }
I20260927 22:14:38.398758 23424 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6a0a601cbb64e239fb817c332faff68 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:14:38.399088 23435 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6a0a601cbb64e239fb817c332faff68 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7683754b03af4cb19dc04fb3815dc76b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7683754b03af4cb19dc04fb3815dc76b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } } peers { permanent_uuid: "f6a0a601cbb64e239fb817c332faff68" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } } peers { permanent_uuid: "1ca15165cd77409daae3d8d3a289bad3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } } }
I20260927 22:14:38.399536 23435 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6a0a601cbb64e239fb817c332faff68 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:14:38.412065 23441 catalog_manager.cc:1259] Loaded cluster ID: 698c79eeb1474f81ba457b954bfe3c89
I20260927 22:14:38.412196 23441 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f6a0a601cbb64e239fb817c332faff68: loading cluster ID for follower catalog manager: success
I20260927 22:14:38.416337 23441 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f6a0a601cbb64e239fb817c332faff68: acquiring CA information for follower catalog manager: success
I20260927 22:14:38.419087 23441 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f6a0a601cbb64e239fb817c332faff68: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 22:14:38.444095 388 mini_kdc.cc:348] Time spent kdestroy: real 0.107s user 0.001s sys 0.025s
W20260927 22:14:38.509687 23301 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:14:38.481152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58226 (local address 127.0.97.62:46257)
0927 22:14:38.481420 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:38.481430 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:38.505397 (+ 23967us) server_negotiation.cc:407] Connection header received
0927 22:14:38.506955 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:38.506964 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:38.507092 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:38.507187 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:38.509349 (+ 2162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58226: BlockingRecv error: recv got EOF from 127.0.0.1:58226 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260927 22:14:38.511500 23304 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:14:38.488164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59380 (local address 127.0.97.61:44909)
0927 22:14:38.488422 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:38.488432 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:38.505991 (+ 17559us) server_negotiation.cc:407] Connection header received
0927 22:14:38.507797 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:38.507807 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:38.507910 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:38.508004 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:38.511156 (+ 3152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59380: BlockingRecv error: recv got EOF from 127.0.0.1:59380 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260927 22:14:38.514860 23377 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:14:38.501662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44642 (local address 127.0.97.60:35689)
0927 22:14:38.501883 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:38.501890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:38.512952 (+ 11062us) server_negotiation.cc:407] Connection header received
0927 22:14:38.513094 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:38.513100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:38.513190 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:38.513298 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:38.514583 (+ 1285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44642: BlockingRecv error: recv got EOF from 127.0.0.1:44642 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:14:38.568352 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:38.548100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59396 (local address 127.0.97.61:44909)
0927 22:14:38.548309 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:38.548317 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:38.559117 (+ 10800us) server_negotiation.cc:407] Connection header received
0927 22:14:38.559759 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:38.559767 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:38.560007 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:38.560161 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:14:38.560971 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:38.562157 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:38.564297 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:38.565852 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:38.565980 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:38.566740 (+ 760us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:14:38.567431 (+ 691us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:14:38.567567 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:38.567840 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 22:14:38.567952 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 22:14:38.612069 23377 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:14:38.602192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44656 (local address 127.0.97.60:35689)
0927 22:14:38.602439 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:38.602446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:38.609613 (+ 7167us) server_negotiation.cc:407] Connection header received
0927 22:14:38.610195 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:38.610204 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:38.610365 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:38.610456 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:38.611772 (+ 1316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44656: BlockingRecv error: recv got EOF from 127.0.0.1:44656 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 22:14:38.655527 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:38.647003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59408 (local address 127.0.97.61:44909)
0927 22:14:38.647217 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:38.647224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:38.648489 (+ 1265us) server_negotiation.cc:407] Connection header received
0927 22:14:38.648698 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:38.648704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:38.648812 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:38.648923 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:14:38.649619 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:38.650765 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:38.652936 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:38.653549 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:38.653686 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:38.654269 (+ 583us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:14:38.654409 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:14:38.654500 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:38.655139 (+ 639us) server_negotiation.cc:299] Negotiation successful
0927 22:14:38.655225 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 22:14:38.694880 23377 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:14:38.685416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44672 (local address 127.0.97.60:35689)
0927 22:14:38.685671 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:38.685678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:38.692812 (+ 7134us) server_negotiation.cc:407] Connection header received
0927 22:14:38.693332 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:38.693338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:38.693435 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:38.693527 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:38.694572 (+ 1045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44672: BlockingRecv error: recv got EOF from 127.0.0.1:44672 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 22:14:38.697531 23407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7683754b03af4cb19dc04fb3815dc76b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7683754b03af4cb19dc04fb3815dc76b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } } peers { permanent_uuid: "f6a0a601cbb64e239fb817c332faff68" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44909 } } peers { permanent_uuid: "1ca15165cd77409daae3d8d3a289bad3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35689 } } }
I20260927 22:14:38.697891 23407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7683754b03af4cb19dc04fb3815dc76b [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:38.755070 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:38.730912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59418 (local address 127.0.97.61:44909)
0927 22:14:38.731106 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:38.731113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:38.747804 (+ 16691us) server_negotiation.cc:407] Connection header received
0927 22:14:38.748089 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:38.748098 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:38.748205 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:38.748304 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:14:38.749016 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:38.750236 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:38.752529 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:38.753226 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:38.753331 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:38.754059 (+ 728us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:14:38.754213 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:14:38.754281 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:38.754524 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 22:14:38.754694 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 22:14:38.801905 23377 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:14:38.792251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44684 (local address 127.0.97.60:35689)
0927 22:14:38.792479 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:38.792492 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:38.799506 (+ 7014us) server_negotiation.cc:407] Connection header received
0927 22:14:38.800133 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:38.800141 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:38.800279 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:38.800380 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:38.801616 (+ 1236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44684: BlockingRecv error: recv got EOF from 127.0.0.1:44684 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 22:14:38.887981 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:38.860606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59432 (local address 127.0.97.61:44909)
0927 22:14:38.860814 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:38.860821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:38.879372 (+ 18551us) server_negotiation.cc:407] Connection header received
0927 22:14:38.879925 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:38.879932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:38.880028 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:38.880094 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:14:38.880802 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:38.881961 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:38.884405 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:38.885105 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:38.885218 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:38.886899 (+ 1681us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:14:38.887111 (+ 212us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:14:38.887194 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:38.887464 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 22:14:38.887581 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 22:14:38.933566 23377 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:14:38.931434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44692 (local address 127.0.97.60:35689)
0927 22:14:38.931736 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:38.931743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:38.932017 (+ 274us) server_negotiation.cc:407] Connection header received
0927 22:14:38.932216 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:38.932222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:38.932342 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:38.932455 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:38.933363 (+ 908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44692: BlockingRecv error: recv got EOF from 127.0.0.1:44692 (error 108)
Metrics: {"server-negotiator.queue_time_us":158}
I20260927 22:14:38.951923 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:38.944796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59448 (local address 127.0.97.61:44909)
0927 22:14:38.944973 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:38.944984 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:38.945476 (+ 492us) server_negotiation.cc:407] Connection header received
0927 22:14:38.945663 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:38.945671 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:38.945766 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:38.945870 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:14:38.946338 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:38.947585 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:38.949468 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:38.950317 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:38.950455 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:38.951043 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:14:38.951229 (+ 186us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:14:38.951325 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:38.951516 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 22:14:38.951605 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 22:14:38.961043 23377 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:14:38.958678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44694 (local address 127.0.97.60:35689)
0927 22:14:38.958865 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:38.958871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:38.959620 (+ 749us) server_negotiation.cc:407] Connection header received
0927 22:14:38.959796 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:38.959802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:38.959901 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:38.960004 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:38.960757 (+ 753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44694: BlockingRecv error: recv got EOF from 127.0.0.1:44694 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:14:38.987758 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:38.981097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59454 (local address 127.0.97.61:44909)
0927 22:14:38.981366 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:38.981376 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:38.981735 (+ 359us) server_negotiation.cc:407] Connection header received
0927 22:14:38.981956 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:38.981960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:38.982032 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:38.982159 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:14:38.982688 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:38.983516 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:38.985570 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:38.986227 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:38.986341 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:38.986971 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:14:38.987142 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:14:38.987294 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:38.987389 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 22:14:38.987468 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260927 22:14:38.996219 23377 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:14:38.994273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44706 (local address 127.0.97.60:35689)
0927 22:14:38.994487 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:38.994494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:38.994863 (+ 369us) server_negotiation.cc:407] Connection header received
0927 22:14:38.995018 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:38.995023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:38.995134 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:38.995256 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:38.995949 (+ 693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44706: BlockingRecv error: recv got EOF from 127.0.0.1:44706 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:14:38.997823 23391 catalog_manager.cc:1259] Loaded cluster ID: 698c79eeb1474f81ba457b954bfe3c89
I20260927 22:14:38.997896 23391 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1ca15165cd77409daae3d8d3a289bad3: loading cluster ID for follower catalog manager: success
I20260927 22:14:39.000447 23391 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1ca15165cd77409daae3d8d3a289bad3: acquiring CA information for follower catalog manager: success
I20260927 22:14:39.002140 23391 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1ca15165cd77409daae3d8d3a289bad3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:14:39.039326 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:39.032480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59456 (local address 127.0.97.61:44909)
0927 22:14:39.032709 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:39.032717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:39.033300 (+ 583us) server_negotiation.cc:407] Connection header received
0927 22:14:39.033539 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:39.033543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:39.033656 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:39.033739 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:14:39.034283 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:39.035192 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:39.037316 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:39.037813 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:39.037905 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:39.038459 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:14:39.038575 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:14:39.038715 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:39.038905 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 22:14:39.038975 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:14:39.052345 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:39.045135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44716 (local address 127.0.97.60:35689)
0927 22:14:39.045347 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:39.045355 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:39.045975 (+ 620us) server_negotiation.cc:407] Connection header received
0927 22:14:39.046098 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:39.046105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:39.046341 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:39.046488 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:14:39.046885 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:39.048047 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:39.049391 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:39.050096 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:39.050247 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:39.050746 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:14:39.051595 (+ 849us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:14:39.051759 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:39.051859 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 22:14:39.051956 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 22:14:39.078990 23527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:39.058371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.0.97.62:46257)
0927 22:14:39.069600 (+ 11229us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:39.069620 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:39.070357 (+ 737us) server_negotiation.cc:407] Connection header received
0927 22:14:39.070663 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:39.070670 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:39.070915 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:39.071119 (+ 204us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:14:39.072003 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:39.073188 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:39.075466 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:39.076163 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:39.076348 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:39.077246 (+ 898us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:14:39.077782 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:14:39.077903 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:39.078475 (+ 572us) server_negotiation.cc:299] Negotiation successful
0927 22:14:39.078610 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11062,"thread_start_us":11257,"threads_started":1}
I20260927 22:14:39.086193 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 23158
I20260927 22:14:39.087275 23220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:14:39.087949 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:39.058720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59458 (local address 127.0.97.61:44909)
0927 22:14:39.058951 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:39.058960 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:39.071208 (+ 12248us) server_negotiation.cc:407] Connection header received
0927 22:14:39.071555 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:39.071565 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:39.071702 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:39.071804 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:14:39.072786 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:39.074337 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:39.082758 (+ 8421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:39.083508 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:39.083606 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:39.086915 (+ 3309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:14:39.087066 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:14:39.087142 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:39.087398 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 22:14:39.087499 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:14:39.089191 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:39.064270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44732 (local address 127.0.97.60:35689)
0927 22:14:39.064467 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:39.064475 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:39.077271 (+ 12796us) server_negotiation.cc:407] Connection header received
0927 22:14:39.077572 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:39.077579 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:39.077705 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:39.077778 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:14:39.078609 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:39.081167 (+ 2558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:39.086644 (+ 5477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:39.087351 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:39.087542 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:39.088285 (+ 743us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:14:39.088437 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:14:39.088511 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:39.088790 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 22:14:39.088900 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:14:39.203845 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 23158
W20260927 22:14:39.224327 23251 connection.cc:535] server connection from 127.0.0.1:40606 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 22:14:39.224753 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 23232
I20260927 22:14:39.225793 23294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:14:39.332273 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 23232
I20260927 22:14:39.353377 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3T1nfn/build/asan/bin/kudu pid 23307
I20260927 22:14:39.354563 23369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:14:39.463176 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3T1nfn/build/asan/bin/kudu with pid 23307
2026-09-27T22:14:39Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4442 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9339 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (366442 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST